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FOREWORD 


C co^c)) 

Th«  tventy-Eighth  Conference  on  the  Design  of  Experiments  in  Amy  Research, 
Development  and  Testing  had  as  its  host  the  U.  S.  Amy  Combat  Development 
Experimentation  Command^  Fort  Ord,  California.  It  was  held  at  the  Hilton  Inn 
Resort,  Monterey,  California,  on  20-22  October  1982.sA  brief  history  of  the 
host  installation  appeared  in  a  booklet  issued  the  attendees  at  this  meeting. 
Exerpts  from  this  booklet  are  reproduced  below. 


A  U.3.  ARMY  COMBAT 
fflk  DEVELOPMENTS 
EXPERIMENTATION 
W  COMMAND 


FORT  ORD, 
CALIFORNIA 


/  COMBAT 
/  DEVELOPMENTS 
I  EXPERIMENTATION 
(j^MAND . .  ^ 

The  United  State#  Army  Combat  Develop-  .... 
ments  Experimentation  Commandlwas  estab¬ 
lished  In  1956  at  Fort  Ord,  Calif.,  setting  the 
stage  for  the  Introduction  of  a  new  form  of  mili¬ 
tary  evaluation ...  the  Combat  Field  Experi¬ 
ment. . .  . - - - 

The  command  Is  more  familiar  to  many  peo-  ^ 
pie,  military  and  civilian  alike,  as  "CDEC.  iWltn 
Its  highly  sophisticated  electronic  field 
laboratory,  located  at  Fort  Hunter  Liggett, 
CDEC's  military-scientific  team  has  the  mission 
of  providing  hard  factual  answers  to  basic  ques¬ 
tions  about  how  the  Army  of  the  future  should 
be  organized,  how  It  should  be  equipped  and 
how  It  can  best  flghKLxD 
To  provide  such  answers  CDEC  has 
developed  a  method  of  evaluation  significantly 
different  frdm  anv<hlng  previously  available  to 
the  military  decision  maker.  CDEC  applies  the 
technique  of  the  scientific  field  experiment  to 
military  systems  and  problems.  Experiments  are 
conducted  under  conditions  simulating  as 
closely  as  possible  those  of  an  actual  combat 
situation. 

CDEC  began  with  soldiers  armed  with  stop¬ 
watches,  compasses,  slide  rules  and  clipboards 
as  data  collection  systems.  Today,  CDEC  has 
evolved  Into  a  high  technology  command  using 
computers,  lasers,  intervisibility  equipment, 
an  accurate  position  location  system  and  other 

sophisticated  measurement  equipment.  - 

r,Vlslon  to  Victory”  Is  CDEC’s  motto.  It  Is  a 
vision  with  a  purpose  that  centers  on  the  pro¬ 
duction  of  hard  date  through  field  experimenta¬ 
tion  and  operations  research  pointing  the  way  to 
the  Army  of  tomorrow. 


CDEC 


CDEC’*  experiment*  Involve  highly  realistic 
mock  battles  during  which  casualties  from 
various  types  of  engagements  are  taken  out  of 
the  battle  by  a  computer  operating  as  a  high¬ 
speed,  Impartial  umpire. 

Hera,  weapons  “fire”  beams  of  light  Instead  of 
live  ammunition.  Each  weapon  Is  equipped  with 
a  low-power,  eye-safe  laser  system  and  each 
soldier-player  or  vehicle,  such  as  a  tank  or 
arlcraft,  has  several  laser  detectors  which  send 
signals  when  “hit”  by  a  laser  beam.  The  players 
and  weapons  are  linked  by  radio  to  a  central 
computer. 

Wner.  a  laser  Is  fired,  a  coded  Impulse  Is  sent 
to  the  computer.  If  the  beam  hits  any  of  several 
laser  detectors  on  the  target,  Its  particular  coded 
Impulse  allows  the  computer  to  determine  in¬ 
stantly  which  weapon  It  came  from,  Its  location 
and  whether  It  was  powerful  and  close  enough  to 
destroy  the  target.  These  Instruments  tell  play¬ 
ers  almost  Instantly  If  they  were  "hit  ’  or 
“missed”  during  these  simulated  flreflghts. 

Computers  take  tanks  and  other  vehicles  out 
of  action  when  hit  by  simulated  fire.  Wbm 
ammo  loads  are  exhausted,  these  same  com¬ 
puterized  systems  can  simply  “turn  off  the 
players’  weapons. 
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The  Army  Mathematics  Steering  Committee  (AMSC),  the  sponsor  of  the  Conferences 
on  the  Design  of  Experiments,  appreciates  the  fact  that  the  Combat  Development 
Experimentation  Command  was  willing,  for  a  second  time,  to  serve  as  host  for 
these  conferences.  For  both  of  these  meetings.  Dr.  Marion  Bryson,  Scientific 
Advisor,  USA  CDEC,  has  served  as  chairman  on  local  arrangements.  His  careful 
planning  helped  moke  these  two  of  the  most  profitable  meetings  in  this  series 
of  statistical  symposia. 

The  Program  Committee  would  like  to  thank  Dr.  Larry  Crow  of  the  U.S.  Army 
Materiel  Systems  Analysis  Agency  for  organising  the  "Special  Software  Test  and 
Evaluation  Session":  and  also  Mr.  Langhorne  Withers,  U.S.  Army  Operational  Test 
and  Evaluation  Agency,  for  arranging  a  special  session  devoted  to  "Logistic 
Supportabllity".  The  agenda  gives  Information  about  these  two  interesting 
solicited  events  for  this  conference.  Members  of  the  Program  Committee  feel 
they  were  fortunate  in  obtaining  the  following  nationally  known  scientists  to 
give  invited  addresses  at  this  meeting. 

SPEAKER  AMD  AFFILIATION 


Professor  Brad  Efrom 
Stanford  University 

Professor  Leo  Brelman 
University  of  Callfornla- 
Berkeley 

Professor  David  W.  Scott 
Rica  University 


Frofeasor  Nancy  R.  Mann 
University  of  Callfornia- 

Los  Angeles 


TITLE  OF  ADDRESS 
Bootstrap  Methods 


Tools  in  Data  Analysis 


Nonparametric  Bivariate  Density 
Estimation  as  a  Tool  for  Data 
Analysis 

The  Influence  of  W.  Edward  Doming 
tM  the  Implementation  of 
statistical  Quality  Control— The 
Early  Days  and  Now 


Another  event  associated  with  this  conference  was  a  tutorial  seminar  on 
"Mon-Parametric  Statistics".  It  was  given,  on  18  -  19  October  1982,  at  the 
U.S.  Army  Combat  Developments  Experimentation  Command  by  Professor  William 
J.  Conover,  Texas  Tech  University. 

The  winner  of  the  second  Wilks  Award  for  Contributions  to  Statistics].  Methodolo¬ 
gies  in  Army  Research,  Development  and  Testing  was  presented  to  Professor 
Bernard  Harris  of  the  Mathematics  Research  Center,  University  of  Wisconsin-Madison, 
at  a  banquet  held  at  the  Naval  Postgraduate  School  on  Wednesday  night,  20 
October  1982.  This  honor  was  bestowed  on  Dr.  Harris  for  his  many  contributions 
to  various  statistical  fields.  He  has  helped  Army  scientists  with  many  of 
their  design  problems,  and  his  advice  in  conducting  these  conferences  haB 
proved  Invaluable.  He  recently  developed, together  with  Dr.  Andrew  P.  Soms, 
new  methodologies  and  optimality  results  for  the  long  unsolved  problem  of 
confidence  bounds  for  system  reliability. 

Members  of  the  AM8C  feel  that  it  is  appropriate  to  again  express  their  thanks 
to  Mr.  Philip  G.  Rust  of  Thomasvllle,  Georgia  for  endowing  both  of  the  Wilks 
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Award*.  The  first  one  entitled,  "The  Samuel  S.  Wilks  Memorial  Medal  and 
Award",  was  initiated  in  1964  and  Is  now  being  administered  by  the  American 
Statistical  Association.  The  second  one,  initiated  in  1981,  is  called  "The 
Wilks  Award  for  Contributions  to  Statistical  Methodologies  in  Army  Research, 
Development  and  Testing",  and  is  under  the  auspices  of  the  AMSC.  Mr.  Rust's 
generous  gifts  in  memory  of  his  friend,  Sam  Wilks,  will  contribute  to  the 
welfare  of  the  military  services  as  well  as  foster  statistical  science  in 
general. 
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ENTROPY  INTERPRETATION  OF  GOODNESS  OF  FIT  TESTS 


Emanuel  Parzen 
Institute  of  Statistics 
Texas  A&M  University 


ABSTRACT.  This  paper  describes  a  synthesis  of  statistical  reasoning 
called  FUOfAT  (because  It  Is  funi  functional  (useful);  based  on  functional 
analysis;  estimates  functions;  and  all  graphs  are  of  functions).  FUN.STAT  has 
three  Important  components:  quantile  and  density* Quantile  signatures  of 
populations,  entropy  and  Information  measures,  ana  functional  statistical 
Inference. 

A  FUN.STAT  approach  to  the  problem  of  Identifying  the  probability 
distribution  F(x)  of  a  random  variable  X  from  a  random  sample  Is  outlined. 

To  Identify  FQ  In  the  location-scale  parameter  model  F(x)  ■  F0((x-u)/c),  wo 

estimate  entropy  difference  a  ■  H°(f)  -  H(f).  H(f)  Is  Shannon  entropy  and 

H°(f)  ■  log  a  +  H(fQ)  Is  entropy  of  the  assumed  model  (which  may  maximize 

entropy).  Estimators  Hj,  Hg,  H3  of  H(f)  are  defined  which  are  respectively 

fully  paramjtrlc,  fully  non- parametric,  and  parametric- select.  Significance 
levels  for  a  are  obtained  by  Monte  Carlo  methods.  The  family  of 
parametric- select  estimators  of  A  may  provide  optimum  tests  of  F  (such  as 
normal  or  exponential)  and  estimators  of  F  when  one  rejects  F0.  0 

KEY  WORDS:  Entropy-based  statistical  inference,  goodness  of  fit  tests, 
test  for  normality,  Shaplro-Wllk  statistic,  quantile,  density-quantile, 
quantile-density,  autoregressive  density  estimator. 


1.  INTRODUCTION.  Let  X1 , . . . ,Xn  be  a  random  sample  of  a  continuous 

random  variable  X  with  distribution  function  F(x)  ■  Pr[X<x],  -«*<x<"»  and 

quantile  function  C|(u)  *  F-1(u),  0<u<l,  Tests  of  normality  or  exponential  1 ty 
are  special  cases  of  a  locatlon-scTli1  parameter  model,  which  we  denote  by  the 
hypothesis 

H0i  F(x)  •  F0(SJL),  q(u)  ■  v  +  c  q0(u) 

where  FQ(x)  Is  a  specified  distribution  with  quantile  function  Q0(u).  Table  1 

lists  F_  and  for  various  standard  distributions, 
o  o 
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Many  statistics  have  been  Introduced  by  statisticians  to  test  the 
composite  (location  and  scale  parameters  unspecified)  hypothesis  of  normality. 
A  superior  omnibus  test  of  ngrmallty  (1nAterms  of  power)  seems  to  be  provided 
by  a  test  statistic  W  ■  02/01  »  where  01  and  02  are  scale  estimators  defined 
as  follows:  01  1$  sample  standard  deviation  while  02  Is  a  linear  combination 
of  order  statistics  estimator  of  o.  We  call  W  a  statistic  of  Shaplro-Wllk 
type  because  It  Is  a  variant  of  a  test  Introduced  by  Shapiro  and  Wllk  (1965) 
and  Shapiro  and  Francla  (1972). 


The  question  arises:  to  discover  a  motivation  for  the  W  statistic  which 
explains  the  source  of  Its  power,  and  to  use  this  Insight  to  extend  W  to 
other  distributions  F  .  In  this  paper  we  propose  that  the  power  of  W  can  be 


explained  by  representing  It  as  an  “entropy  difference"  test  statistic.  We 
show  that  the  test  statistic  for  normality  Introduced  by  Vaslcek  (1977)  Is 
also  an  entropy  difference  statistic,  as  are  test  statistics  Introduced  In 
Parzen  (1979). 

2.  INFORMATION  DIVERGENCE  AND  ENTROPY.  To  compare  two  distribution 
functions  t(x)  and  G(x)  with  probability  densities  f ( x)  and  g(x),  a  useful 
measure  Is  Information  divergence,  defined  by 


-  /Tj-log  f(x)  dx 
It  can  be  decomposed  Into  cross-entropy 

H(f-.g)  •  g(x)>  f(x)  dx 

and  entropy 

H(f)  -  H(f;f)  ■  /r„{-log  f(x)>  f(x)  dx 
by  the  Important  Identity 

0  <  I ( f ; g )  -  H(fig)  -  H(f). 

To  estimate  entropy  It  Is  useful  to  express  It  In  terms  of  the  quantile 
density  function  q(u)  and  density-quantile  function  fQ(u)  defined  by 

q( u)  -  Q' (u) .  fQ(u)  ■  f(Q(u>)  -  (q(u)f1 
By  making  the  change  of  variable  u  -  F(x)  one  can  show  that 

H(f)  -  fl  -  log  fQ(u)  du 
■  /I  log  q(u)  du. 


3'*  *T 


Under  the  hypothesis  HQ  that  F(x)  «  Fq({x-vi)/o)»  a  location-scale  model, 
q(u)  ■  oq0(u)  and 

H(f)  -  log  o  +  H(fQ). 

.3.  ENTROPY  DIFFERENCE  TO  TEST  GOODNESS  OF  FIT.  To  test  the  hypothesis 
HQ  we  propose  to  Investigate  (and  eventually  establish  how  to  use  optimally) 

test  statistics  which  are  entropy- difference  statistics 
A(f)  ■  H°(f)  -  H(f) 

where  H°(f)  Is  a  parametric  evaluation  of  the  entropy  of  f,  evaluated  under 
the  assumption  that  It  obeys  HQl  defined  by 

H°(f)  •  log  o  +  H(f0), 

while  H(f)  Is  a  non-parametric  evaluation  of  H(f),  usually  most  conveniently 
obtained  by 

H(f)  -  log  q(u)  du  . 

To  estimate  H(f)  we  have  three  types  of  estimators  which  we  call 
fully  parametric  estimator, 
fl2  fully  non-parametric  estimator, 

H3  smooth  or  parametric  select  estimator 

Similarly  to  estimate  H°{f)  we  have  several  types  of  estimators  depending  on 
the  estimator  oj  we  adopt  for  c;  thus 

-  log  Sj  +  H(fQ) 

Three  Important  possibilities  for  oj  are: 

01  maximum  likelihood  estimator, 

a2  optimal  linear  combination  of  order  statistics  estimator 
o g  estimator  of  score  deviation  oj  ■  /J  1*0Q0(u)  q( u)  du. 

Under  HQ  these  estimators  are  all  asymptotically  efficient  estimators  of  d'. 
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While  one  can  conceive  of  about  9  possible  estimators  of  the  entropy 
difference  A,  we  discuss  only  three  estimators  which  we  denote  An,  &12, 

and  A33  . 


4.  ENTROPY-DIFFERENCE  INTERPRETATION  OF  SHAPIRO-WILK  STATISTIC 
To  test  the  hypothesis  HQ:  X  Is  N(p,o4),  a  test  statistic  W  of 

Shaplro-Wllk  type  is  of  the  form 
W  *  o2  ♦  oj 

where  01  Is  the  sample  standard  deviation  and 


Is  an  asymptotically  efficient  estimator  of  a  based  on  linear  combinations 
of  the  order  statistics  X^j<  ...<X^nj  of  the  random  sample.  The  first  step 

In  the  entropy  Interpretation  of  W  Is  to  consider  Instead  the  statistic 
An  ■  -  log  W  ■  log  01  -  log  02  •  H°  -  M, 
wh«rt  [with  f„(x)  •  <h(x)  (2.)'1*  exp  -(%)  x*.  and  H(fQ)  -  1  (1  +  log  2.)] 

H°  *  log  oi  +  H(f0) 

Is  an  estimator  of  H°(f)  based  on  oi,  and  Is  a  purely  parametric  estimator 
of  H(f)  based  on  the  parametric  estimator  o2;  note  Hi  ■  H§. 

Significance  levels  for  the  entropy-difference  statistic  An  «  -  log  W 
are  obtainable  from  tables  of  the  W  statistic  [for  example,  Fill Iben  (1975)]. 
An  example  of  5X  significance  levels  (for  accepting  normality)  are 

A 

An  *  0.05,  for  sample  size  n  ■  20  ; 

* 

An  i  0.023,  for  sample  size  n  -  50 


5.  ENTROPY-DIFFERENCE  INTERPRETATION  OF  VASICEK  STATISTIC 
Yo  test  the  hypothesis  H0:  )(  Is  N(yicr*)  Vaslcek  (1977T proposes  a 
statistic  which  Is  equivalent  to 

A  A  A  A 

Aia  ■  -  H2 

where  H°  Is  an  estimator  of  the  parametric  evaluation  H°(f)  of  entropy,  and 


A 

Ho  Is  a  fully  non-parametrlc  estimator  of  H(f)  based  on  the  gap  or  leap  (of 
order  2v»)  estimator 

%(rvM  *  Tv  {X(j+v)  “  X(j-v)}  ’  j*v+1 . n"v 

of  q(j/(n+l )) ,  and 

A  1  n-V  at  J 

H* '  dx  i-U  %  ^ 

A 

Some  significance  levels  of  A12  are  given  In  Table  2;  they  are  transformations 

of  the  significance  levels  given  by  Vaslcek  (1977)  and  obtained  by  Monte-Carlo 
simulation. 

6.  ENTROPY- DIFFERENCE  INTERPRETATION  OF  PARZEN  GOODNESS  OF  FIT  PROCEDURE 

To  test  the  general  hypothesis  H  :  X  Is  F0(~^-)t  Parzen  (1979)  proposes 
forming  raw  estimators  3(u)  of  0  00 

d(u)  ■  1-  f  Q  (u)  q(u)  , 
ao  0  0 

1  •*  ■» 
where  oQ  ■  fQQ0(t)  q(t)  dt.  To  form  d(u)  and  oQ  we  replace  q(u)  by  the 

least  smooth  gap  estimator  q2  (u).  Smooth  estimators  dm(u)  of  d(u)  are 

formed  by  the  autoregressive  method.  From  estimators  of  the  pseudo-correlations 

p(v)  "  /q  e2ir*uv  d(u)  du,  v-0,+1 , . . . ,+m 

one  estimates  the  coefficients  of  the  autoregressive  order  m  approximator 

d»  •  K.  I  1  +  <l(1)  e^ff1u  t...+a» 
m  m  m  m 

to  d(u).  The  coefficient  K_  pl^ys  an  Important  role  In  entropy  calculations 
since  m 

/J  -  lo9  dm(u)  du  -  -log  ^ 

A  1  A 

can  be  regarded  as  an  estimator  a33  ■  }Q  -  log  d(u)  du  of  A. 

This  formula,  which  we  prove  below,  provides  an  entropy-difference 
Interpretation  of  the  goodness  of  fit  procedures  In  Parzen  (1979). 

To  prove  this  Interpretation  of  A33l  write 
-  log  d(u)  -  log  oQ  -  log  f0Q0(u)-log  q(u) 
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Therefore 


/’  -  I09  d(u)  du  -  H°(f)  -  H(f) 


Is  an  entropy- difference. 

The  autoregressive  estimator  d  (u)  of  d(u)  provides  a  parametric 
select  estimator  of  q(u)  by  m 

q(u)  *  °o 


A  parametric  select  estimator  of  H(f)  Is 
H3  ■  ll  lo9  du 
■  /J  lo9  du  +  H® 

where 

H$  "  log  J0  +  H(f0) 

a.  W 

Is  an  estimator  of  H  (f)  based  on  aQ. 

a 

The  parametric  select  entropy-difference  test  statistic  A33should  be 

A 

denoted  a33  because  It  depends  on  the  order  m  of  the  autoregressive 

estimator  3m(u)  of  d(u).  Significance  levels  of  a33  derived  by  a  very 

approximate  Monte  Carlo  simulation  (In  the  case  of  testing  for  normality) 
are  given  In  Table  2.  They  show  that  the  parametric  select  estimators  of  A 
provide  a  smooth  progression  of  significance  levels  from  the  fully  parametric 
estimators  of  A  to  the  fully  non- parametric  estimators.  In  practice,  we 
recommend  adaptive  determination  of  the  order  m  by  the  data,  rather  than 
choosing  a  fixed  order  m. 


It  may  be  useful  to  use  a  rough  approximation  to  the  5*  significance 


levels  of  a33  _  which  Is  provided  by  2m/ n. 
,s  F0(**>  1,1 


A  criterion  for  accepting  H  :  X 


*33 


.m 


-  log  Km  <  ^  «  m-1,2,... 


1 

One  rejects, HQ  If  there  exists  a  value  of  m  for  which  the  Akalke-type  criterion 
Aic(m)  •  nj  +  log  i^,  <  o  i 


the  value  of  m  which  minimizes  AIC(m)  is  chosen  as  an  "optimal"  value  m. 

An  optimal  parametric- select  estimator  of  the  true  quantile-density  function 
q(u)  is 

% (u)  *  K  ^(u)  q°(u)  • 


7.  CONCLUSION 

We  believe  that  the  interpretation  given  in  this  paper  of  powerful 
goodness  of  fit  procedures  as  entropy-difference  statistics  provides  a 
striking  demonstration  of  the  FUN.STAT  synthesis  of  statistical  reasoning. 
In  addition  to  elegance  of  the  theory,  very  practical  and  Implementable 
procedures  are  obtained. 


The  parametric  select  estimators  a33  of  entropy-difference  test 

statistics  for  goodness  of  fit  have  for  m-l  approximately  the  properties  of 
fully  parametric  estimators  (such  as  Shaplro-Wllk  En)  and  have  for  large 
values  of  m  approximately  the  properties  of  fully  ngn- parametric  estimators 
(such  as  Vaslcek  Ai2).  Thus  It  appears  the  series  a33  m  provide  all  the  test- 
statistics  required.  Further  the  autoregressi ve  approich  provides 
non- parametric  estimators  of  the  true  distribution  when  one  rejects  the  null 
hypothesis  H  • 


One  may  find  that  a  sample  passes  the  goodness  of  fit  procedure  for  two 
null  hypotheses.  An  appealing  procedure,  whose  properties  remain  to  be 
Investigated, Is  to  choose  that  null  hypothesis  for  which  a33  Is  always  less 
than  the  corresponding  statistic  for  the  other  hypothesis. 

The  entropy-difference  statistics  A33  are  Implemented  In  our  one-sample 

univariate  data  analysis  computer  program  6nESAM.  Table  3  lists  auto¬ 
regressive  estimates  of  entropy-difference  when  testing  for  normality  data 
sets  In  Stigler  (1977).  An  asterisk  Indicates  a  data  set  which  Is  not 
normal  In  our  judgement. 

In  Table  2  we  report  significance  levels  for  a12  obtained  (by  Monte  Carlo 
calculations)  by  Dudewicz  and  van  der  Muelen  (1981)  in  the  case  of  testing  for 
uniformity  rather  than  normality. 


The  closeness  of  the  Dudewicz-van  der  Muelen  levels  to  the  Vaslcek 
levels  suggests  a  conjecture,  which  remains  to  be  proved,  that  the  entropy- 
difference  statistics  have  distributions  which  are  approximately  the  same 

for  all  null  hypotheses  HQ:  X  Is  F0(—j^). 


A  final  noteworthy  feature  Is  that  the  autoregressive  method  of 
estimating  quantile-densltv  functions  and  density-quantile  functions, 
Introduced  In  Parzen  (1979),  can  be  shown  to  have  a  maximum  entropy 
property  [compare  Parzen  (1982)]. 
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Trble  2.  5*  SIGNIFICANCE  LEVELS  FOR  ENTROPY  DIFFERENCE  STATISTICS 


Accept  H0*.  X  Is  N(w,o2)  for  some  m  and  o  If  entropy  difference  Is  less  than 
threshold  given. 


Sample 
Size  n 

A 

An 

Shapl ro- 
Wllk 

A 

,  A33,m 

Autoregressive  order  m 
Monte  Carlo  SX  level 
(rough  approximation  2m/n) 

■ - -  ■■  ’A  - -  -  — 

Al2 

Vaslcek  gap  estimator  qk( 
(Dudewlcz-van  der  Muelenr 

v*5  v*4  v»3  v*2  %>■! 

m-l 

m«Z 

m«3 

m»4 

20 

.05 

.141 

.235 

.299 

.378 

.398 

.40 

.40 

.43  .61 

(.10) 

(.20) 

(.30) 

(.40) 

(.50) 

(.43 

.43 

.47  .66) 

SO 

.023 

.045 

.081 

.126 

.153 

.176 

.21  .21 

.23 

(.04) 

(.08) 

(.12) 

(.16) 

(.20) 

(.22  .22 

.24) 

Shapiro-WIlk  and  Vaslcek  levels  are  based  on  Monte  Carlo  simulation  of  normal*, 
Dudewlcz-van  der  Muelen  levels  are  based  on  Monte  Carlo  simulation  of  uniform. 


One  can  conjecture  a  relation  between  gap  order  2v  and  autoregressive  order 
m  for  the  corresponding  estimators  to  nave  similar  distributions  and  therefore 
similar  significance  levels: 

(2v)  m  *  n  ■  sample  size 

To  understand  what  this  conjecture  Is  alleging  note  that  for  n»20,  m«4  Is 
similar  to  2\>  *  6;  for  n»50,  m«6  Is  similar  to  2v  ■  8. 

When  one  uses  gap  estimators  of  q(u),  and  thus  of  entropy,  one  has  the 
problem  of  determining  the  order  2v.  One  can  more  easily  develop  criteria 
for  determining  the  order  m  of  autoregressive  estimators  of  q(u). 
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ABSTRACT.  Let  ttiere  be  given  c  independent  random  samples  of  continuous 
random  variables  of  size  n^,  n2,  ...  ,nc;  and  denote  the  observations  in  the 
ith  sample  by  xi2,  •••  »  xln  .  Suppose  it  is  desired  to  test  the  null 
hypothesis  that  the  samples  all  c&me  from  the  same  population. 

Bimbaum  and  Hall  proposed  for  this  null  hypothesis  the  test  statistic 

DCn^nj,  ...  ,nc)  -  Sup  |  f[(x)  -  fJ(x)| 

i » j 

* 

for  i,j  >  1,2,  ...  ,c,  where  F,  denotes  the  empirical  cumulative  distribution 
th  * 

function  for  the  i  sample.  In  their  1960  paper  they  published  the  proba¬ 
bilities  P[D(n,n,n)  <  r]  for  n  «  1(1)20(2)40  where  r  ■  k/n,  k  ■  1,2,  ...,n. 

The  tables  referenced  here  are  an  extension  of  the  Bimbaum-Hall  Tables, 
resulting  from  the  examination  of  a  larger  number  of  samples  and  the  considera¬ 
tion  of  unequal  sample  sizes.  In  addition,  an  application  of  this  work  to  a 
problem  in  ballistics  is  discussed.  Although  length  precludes  the  inclusion 
of  the  tables  in  this  paper,  they  are  available  in  a  technical  report  published 
by  the  Ballistic  Research  Laboratory  under  the  same  title. 

I.  INTRODUCTION.  Let  there  be  given  c  independent  random  samples  of  con¬ 
tinuous  random  variables  of  size  „  n2,  ...  ,nc,  where  n^  +  n2  +  . . .  +  nc  ■  N; 
and  denote  the  observations  in  the  ith  sample  by  x^,  xi2„  ...  ,  x^n  .  Suppose 
it  is  desired  to  test  the  null  hypothesis  that  the  samples  all  come  trom  the 
same  population. 

Bimbaum  and  Hall*  proposed  for  this  null  hypothesis  the  following 
two-sided  and  one-sided  test  statistics  respectively: 

V - 

Bimbaum,  Z.W.  and  Hall,  R.A.,  "Small  Sample  Distributions  for  Multi-Sample 
Statistics  of  the  Smirnov  Type,"  The  Annals  of  Mathematical  Statistics, 

Vol.  31,  No.  3,  pp.  710-720,  September  1960. 


13 


.  th 


i,j  ■  1,2,  ...  ,c,  where  denotes  the  empirical  c.df  for  the  i  sample. 


11.  COMPUTATIONAL  METHODS.  Under  the  null  hypothesis,  the  c  samples  may 

be  considered  as  c  successive  drawings  of  n^,n2»  ...  ,nc  observations  from  the 

same  population,  with  equal  probability  of  each  of  the  N!  ways  of  drawing  the 

ordered  sample  of  site  N.  The  values  taken  on  by  the  random  vector 
*  *  ■  * . 

IFj(x) ,F^(x),  ...  ,Fc(x)],  for  fixed  x,  after  a  component-wise  transformation 
k^  "  niFi(x)’  establish  *  one-to-one  correspondence  between  the  combined  ordered 
samples  and  a  path  in  c-space  from  (0,0,  ...  ,0)  to  (n^ng, 
of  distinct  combined  ordered  samples  (paths)  is  Nl/(n. !ng!  . 
which  is  equally  likely  to  occur  under  the  null  hypothesis. 

Introducing  the  notation 


»V* 


The  number 


ncl),  each  of 


Qfk^kg,  ...  ,kc)  ■  number  of  paths  from  (0,0,  ...  ,0)  to  (kj.kg,  ...  ,kc) , 
the  following  difference  equation  is  established: 


Q(kj,kg,  ...  ,kc)  ■  Q(k|-l,kgi  ...  *k^)  +  Qtk^kg'l,  •••  »kc)  +  ••• 

*  Q(kj,kg,  ...  ,kg-l) 

with  initial  condition  Q(0,0,  ...  ,0)  >  1. 

For  a  given  subset  R  c  R  ■  ((k^kg,  ...  ,kc) |0  <  k^  <  n^,  i»l,2,  ...  ,c), 
let  Qik^.kg,  ...  ,kc;R)  -  number  of  paths  from  (0,0,  ...  ,0)  to  (k^kg,  ...  ,kc) 

not  containing  points  of  R. 

And,  as  beforo 


Q(kj,kg,  ...  ,kg}ft)  ■  Q(kg-l,kg,  ...  ,k^jR)  +  QCk^kg-l,  ...  ,k^jft)  +  ... 

*  Q(kj,kg,  ...  ,kc-l jR) 


with  conditions 


QCkjikj*  '**  »kg}R)  ■  Oj  for  (k^kji  »k^)  6  R. 

Under  the  null  hypothesis,  the  probability  that  the  samples  determine  a 
path  from  (0,0,  ...  ,0)  to  (n^,n2,  ...  ,nc)  which  does  not  encounter  any 
point  of  R  is 

Pj^  ■  Q(nj,n2,  ...  ,ngjR)/[NI/(nj Injl  ...  n ^ I ) ] . 

If  our  decision  rule  is  to  reject  the  null  hypothesis  whenever  the  samples 
determine  a  path  containing  points  in  a  given  set  R,  then  1  -  P^  is  the  probe 
bility  of  an  error  of  the  first  kind. 

The  regions  of  rejection  for  the  two-sided  and  one-sided  tests,  are 
respectively 

DCn^nj,  ...  ,n0)  >  r  and  0+(n1,n2i  ...  ,nc)  >  r, 
which  determine  the  corresponding  sets  R,  R+ 


R  ■  { (kj ,kj , 

»*  .  {(k,,k2. 


see 


•  e  e 


,k  )|  Sup  |  n.k, 

C  i.J  3  1 

,k  )|  Sup  (n.k. 
c  i<j  3  1 


•  "ikjl  ”  Vjr)- 

-  “ikj>  ” 


where  i,j  ■  1,2,  ...  ,c. 


The  tables  were  computed  by  evaluating  the  difference  equation  for 
Q(k^,k2,  ...  ,kc;ft) . 


For  sample  sises  not  included  in  the  tables,  the  inequality 


PfDC^.nj,  ...  ,nc)  <  r]  -  P[  Sup  |  fJ(x)  -  pJ(x)|<  r] 

x,i,  j 

■  1  -  P[Sup|F?(x)  -  f!(x)J  >  r  for  some  i  <  j] 
x  1  3 

^1-1  I...I  P[D(n.,n. ,  ...  ,n.)  >  r],  k  <  c, 
i'<j<..^«  1  3  11 
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•Hows  the  tabled  values  for  both  equal  and  unequal  samples  to  be  used  to 
test  the  null  hypothesis.  The  test  will  be  conservative,  but  should  prove 
useful  for  the  range  of  values  of  c  for  which  it  would  likely  be  applied. 

Asymptotic  expressions  are  not  available  for  extension  of  these  tables. 
As  •  matter  of  fact,  Hodges*  points  out  that  asymptotic  expressions 
advanced  for  the  case  c  ■  2  are  inaccurate  to  an  extent  that  their  usefulness 
is  questionable# 


III.  EXACT  TABUS.  In  their  paper,  Bimbaum  and  Hall  published  the  proba¬ 
bilities  P[D(n,n,n)  <  r]  for  n  ■  1(1)20(2)40  and  r  ■  k/n,  k  ■  1,3,  ...  ,n,  such 

i 

that  the  probabilities  for  each  n  range  from  less  than  ,.90  to  more  than  .995. 

The  tables  referenced  here  are  an  extension  of  the  Bimbaum-Hall  tables. 

The  first  table  contains  the  probabilities  P[D(n,  ...  ,n)  <  r)  with  the  number  of 
Independent  samples  of  sise  n  ranging  from  threa  to  seven  and  the  correspond¬ 
ing  sample  sises  taking  on  values  from  sixty  for  the  three-sample  case  to 
five  for  the  seven-sample  case.  Bimbaum  and  Hall's  values  are  a  subset  of 
the  first  table  as  indicated  in  Figure  1. 


Perhaps  of  more  importance,  the  second  table  contains  the  quantiles  that 
allow  the  test  to  be  applied  to  samples  of  unequal  fixe,  an  option  presently 
not  available.  Here  for  the  useful  cases  c  ■  3,  2  <  »n2*n3  <  25  *nd 

c  ■  4,  2  <  n,,n2,n3,n4  <  15,  are  tables  P[D(n^,  ...  ,nc)  <  r]  for  r  ■  0. 1,0.2, 
...  ,1.0  as  represented  in  Figure  2. 


Figure  2 


Hodges,  J.L.,  Jr.,  "The  Significance  Probability  of  the  Smirnov  Two-Sample 
Test,"  Arkiv  fur  Mathematik,  Vol.  3,  1957.  pp.  469-486. 


IV.  EXAMPLE.  For  purpose  of  illustration,  consider  four  independent 
random  samples,  of  sixes  five,  seven,  seven,  and  thirteen  respectively. 

Sample  I  -0.927,  -2.243,  0.815,  -0.341,  -0.250 

Sample  II  -0.451,  -1.516,  -1.447,  0.504,  1.645,  0.022,  1.098 

Sample  III  0.032,  -1.772,  1.049,  -0.073,  -2.053,  -1.123,  0.799 

Sample  IV  0.250,  -0.185,  -0.028,  0.004,  -0.462,  -0.032,  0.279,  -1.053 

0.597,  0.235,  0.510,  1.103,  0.241 

The  null  hypothesis  to  be  tested  is  that  all  four  samples  come  from  the  same 
distribution.  Evaluation  of  the  statistic  D(5,7,7,13),  ordinarily  programmed 
for  machine  calculation,  can  be  carried  out  in  the  following  manner: 

1)  Pool  and  order  the  samples 


-2.243,  -2.053, 

-1.772,  -1.516,  -1,447, 

-1.123, 

-1.053,  -0.927 

-0.462,  -0.451, 

-0.341,  -0.250,  -0.185, 

-0.073, 

-0.032,  -0.028 

0.004, 

0.022, 

0.032,  0.235,  0.241, 

0.250, 

0.279,  0.504 

0.510, 

0.597, 

0.799,  ‘  0.815,  1.049, 

1.098, 

1.103,  1.645 

* 

2)  For  every  x  in  the  pooled  sample,  evaluate  F.(x),  i  >  1,2, 3, 4,  and 

A  A  “ 

calculate  Sup  Jf^x)  -  F^(x)|,  i,j  ■  1, 2,3,4 
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a"4  fc**  ' '  '\ 


r- 1 


■y-  y\  .”  7\.  Ty.’,’^ATv'.H.T'.y'.T' 


X 

fJ(x) 

V*) 

F4  00 

Sup 

-2.243 

1/S 

0 

0 

0 

0.20 

-2.053 

1/5 

0 

1/7 

0 

0.20 

-1.772 

1/5 

0 

2/7 

0 

0.29 

-1.516 

1/5 

1/7 

2/7 

0 

0.29 

-1.447 

1/5 

2/7 

2/7 

0 

0.29 

-1.123 

1/5 

2/7 

3/7 

0 

0.43 

-1.053 

1/5 

2/7 

3/7 

1/13 

0.35 

-0,927 

2/5 

2/7 

3/7 

1/13 

0.32 

-0.462 

2/5 

2/7 

3/7 

2/13 

0.28 

-0.451 

2/5 

3/7 

3/7 

2/13 

0.28 

-0.341 

3/5 

3/7 

3/7 

2/13 

0.45 

-0.250 

4/5 

3/7 

3/7 

2/13 

0.65 

-0.185 

4/5 

3/7 

3/7 

3/13 

0.57 

-0.073 

4/5 

3/7 

4/7 

3/13 

0.57 

-0.032 

4/5 

3/7 

4/7 

4/13 

0.49 

-0.028 

4/5 

3/7 

4/7 

5/13 

0.42 

0.004 

4/5 

3/7 

4/7 

6/13 

0.37 

0.022 

4/5 

4/7 

4/7 

6/13 

0.34 

0.032 

4/5 

4/7 

5/7 

6/13 

0.34 

0.235 

4/5 

4/7 

5/7 

7/13 

0.26 

0.241 

4/5 

4/7 

5/7 

8/13 

0.23 

0.250 

4/5 

4/7 

5/7 

9/13 

0.23 

0.279 

4/5 

4/7 

5/7 

10/13 

0.23 

0.504 

4/5 

5/7 

5/7 

10/13 

0.09 

0.510 

4/5 

5/7 

5/7 

11/13 

0.13 

0.597 

4/5 

5/7 

5/7 

12/13 

0.21 

0.799 

4/5 

5/7 

6/7 

12/13 

0.21 

0.815 

5/5 

5/7 

6/7 

12/13 

0.29 

1.049 

5/5 

5/7 

7/7 

12/13 

0.29 

1.098 

5/5 

6/7 

7/7 

12/13 

0.14 

1.103 

5/5 

6/7 

7/7 

13/13 

0.14 

1.645 

5/5 

7/7 

7/7 

13/13 

0.00 

3)  Choose  tho  largest  number  appearing  in  the  final  column;  that  is 
D(5,7,7,13)  ■  0.65. 

A 

The  critical  level  a  is  defined  as  the  smallest  significance  level  at  which 
the  null  hypothesis  would  be  rejected  for  the  given  observations.  The  second 
table  shows  that  if  the  four  samples  were  from  the  same  population, 

P[D(5,7,7,J.3)  <  .6]  ■  0.701271  and  P[D(5,7,7,13)  <  .7]  ■  0.827140. 

Therefore,  P[D(5,7,7,13)  >  .6]  ■  0.298729  and  P[D(5,7,7,13)  >  .7]  ■  0.172860. 
Since,  in  this  example,  D(5,7,7,13)  ■  0.65,  a  would  be  between  0.18  and  0.30. 

V.  APPLICATION.  The  test  found  application  in  a  bomblet  study  conducted 
at  the  Ballistic  Research  Laboratory.  In  this  study,  bomblets  were  supplied  by 
three  different  vendors.  Fifteen  bomblets  from  one  vendor  were  filled  with  a 
high  explosive;  thirty  bomblets  supplied  by  each  of  the  other  two  vendors  con¬ 
tained  an  inert  substance.  All  bomblets  were  subjected  to  the  same  field  test, 
meant  to  simulate  the  dispensing  of  a  bomblet  by  a  type  of  munition;  this 
resulted  in  an  out-of-round  characteristic.  A  measurement  was  then  taken  in 
order  to  determine  the  degree  of  ovalness.  The  experimenter  wanted  to  estab¬ 
lish  whether  bomblets  from  all  three  vendors,  filled  with  two  different  materials, 
achieved  approximately  the  same  degree  of  ovalness. 

This  is  tantamount  to  determining  whether  three  independent  samples  come 
from  the  same  population.  Thus,  we  were  able  to  state  a  hypothesis,  evaluate 
the  two-sided  test  statistic,  compare  it  with  the  tabled  values,  and  determine 
whether  or  not  to  reject  the  hypothesis.  The  data  and  analysis  follow. 

Ho:  All  three  samples  of  bomblets  achieve  the  same  degree  of  ovalness. 

Hj!  At  least  one  sample  of  bomblets  achieves  a  significantly  different 
degree  of  ovalness. 
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The  data  (measurement  of  ovalness)  are  as  follows: 


Vendor  I 

0.002 

0.010 

0.036 

0.004 

0.006 

0.038 

0.023 

0.003 

0.013 

0.002 

0.013 

0.010 

0.013 

0.008 

0.007 

Vendor  11 

0.007 

0.004 

0.003 

0.010 

0.009 

0.005 

0.011 

0.002 

0.011 

0.007 

0.010 

0.011 

0.023 

0.009 

0.029 

0.024 

0.004 

0.020 

0.011 

0.019 

0.004 

0.014 

0.011 

0.008 

0.009 

0.014 

0.011 

0.016 

0.007 

0.013 

Vendor  III 

0.007 

0.001 

0.072 

0.013 

0.001 

0.025 

0.007 

0.010 

0.028 

0.004 

0.002 

0.014 

0.020 

0.041 

0.018 

0.011 

0.010 

0.007 

0.035 

0.001 

0.015 

0.008 

0.008 

0;007 

0.007 

0.047 

0.015 

0.054 

0.010 

0.020 

The  test  statistic  takes  on  the  value  0(15,30,30)  ■  0.2333.  Quantiles  of 
0(15,30,30)  are  not  tabulated,  and  so  the  computer  program  which  generated 
the  table  was  rerun  to  obtain  exact  values.  The  results  showed  that 
P[D(15,  30,  30)  <  .2]  -  0.148915  and  P[D(15,  30,  30)  <  .3]  -  0.650568. 
Therefore,  the  critical  level  a  would  be  between  0.35  and  0.86. 

The  data  do  not  suggest  rejecting  the  null  hypothesis  at  a  moderate 
significance  level.  Therefore,  the  experimenter  concluded  that  the 
three  bomblet  samples  achieved  approximately  the  same  degree  of  ovalness. 


REFERENCES 


Becker,  M.P.  and  Taylor,  M.S.,  Quantiles  of  Multi-Sample  Smirnov  Type 
Statistics,  Technical  Report  ARBRL-TR-02331,  Ballistic  Research  Laboratory, 

June  1$81. 

Bimbaum,  Z.W.  and  Hall,  R.A.,  "Small  Sample  Distributions  for  Multi-Sample 
Statistics  of  the  Smirnov  Type,"  The  Annals  of  Mathematical  Statistics. 

Vol.  31,  No.  3,  September  1960. 

Conover,  W.J.,  Practical  Nonparametric  Statistics,  John  Wiley  6  Sons,  Inc., 

New  York,  1971. 

Hodges,  J.L.,  Jr.,  "The  Significance  Probability  of  the  Smirnov  Two-Sample  Test 
Arkiv  fUr  Matematlk,  Vol.  3,  1957. 


AD  P  001574 


ANALYZING  n  SAMPLES 
OF  2  OBSERVATIONS  EACH 


J.  R.  Knaub,  Jr.,  L.  M.  Grlle,  G.  Petet 
II.  S.  Army  Logistics  Center,  Ft.  Lee»VA 


ANALYZING  n  SAMPLES  OF  2  OBSERVATIONS  EACH 

I.  INTRODUCTION  (Modified  Westenberg  Tests  and  Length  of  Initial  Run  (LIR) 

Test)  (Knaub) 

II.  Mathematical  theory  of  LIR  Test  (Knaub). 

III.  Example  (Knaub). 

Appendix  I.  FORTRAN  Code  for  Modified  Westenberg  Tests  (Designed  for 
n  pairs  of  observations)  (Knaub) 

Appendix  II.  Power  Tables  for  LIR  Test  (Knaub) 

Appendix  III.  Discussion  of  Automated  LIR  Test  (ALIRT)  (Petet,  Grlle) 

Appendix  IV.  Acknowledgements 

Appendix  V.  References 


I.  Introduction. 

When  data  are  scarce.  It  Is  common  to  combine  small  samples  from  a 
number  of  sources  considered  to  be  reasonably  similar.  When  sample  sizes  are 
extremely  small,  testing  the  assumption  of  similarity  of  sources  Is  often  only 
attempted  by  subjective  means.  This  paper  provides  a  method  to  add  quantita¬ 
tive  risk  assessments  to  the  study  of  this  assumption,  using  two  observations 
per  sample. 

In  addition  to  general  compatibility  testing  of  the  sources  using 
modified  versions  of  Westenberg's  Interquartile  Range  Test,  and  the  Westenberg-Mood 
Median  Test,  a  new  hypothesis  test  has  been  developed  to  aid  In  Identi¬ 
fying  whether  one  (or  more)  of  the  sources  of  data  provides  a  substantially 
larger  or  smaller  set  of  values  due  to  Its  underlying  population. 

Because  the  probabilistic  risk  assessments  provided  here  address  a  / 

situation  so  commonly  found  in  analyzing  military  operations  as  well  as  In  test  / 

and  evaluation,  details  are  provided  to  simplify  the  Implementation  of  this 
methodology.  A  major  goal  here  has  been  that  power  analyses  be  described  In 
terms  meaningful  to  the  user  and  the  decision  maker.  The  new  hypothesis  test 
makes  use  of  simulation-aided  power  analyses,  f—-- - ...... - __ 

The  tests  for  general  comparability,  using  modified  Westenberg  tests, 
were  first  introduced  In  reference  2.  The  FORTRAN  code  for  tnese  tests  Is 
given  here  In  Appendix  I.  This  program  Involves  straight  forward  binomial 
probabilities.  The  first  format  statement  explains  the  variables  and  the  null 
and  alternative  hypotheses.  It  Is  written  In  terms  of  the  Interquartile  range 
of  the  combined  sample,  but  could  easily  be  written  In  terms  of  being  above  or 
below  the  median  of  the  combined  sample.  The  null  hypothesis  (HQ)  and  the 

alternative  (Hj)  are  repeated  here.  "H0:  Each  sample  has  at  least  100  x  RA 

percent  chance  of  having  one  observation  Inside  and  one  observation  outside  the 
interquartile  range.  H^:  Each  sample  has  at  least  100  x  RB  percent  chance  of 

having  both  observations  fall  together  either  Inside  or  outside."  The  null 
hypotheses  for  the  Westenberg  tests  thus  Indicate  a  general  compatibility  among 
the  data  sources.  If  a  set  of  data  made  up  of  pairs  of  observations  from  a 
number  of  sources  appear  to  be  reasonably  homogeneous  as  judged  by  these  tests, 
however,  It  may  still  be  that  one,  or  perhaps  a  few  pairs  of  observations  may 
have  been  drawn  from  a  source  very  different  from  the  underlying  population  for 
the  majority  of  the  pairs  of  observations.  Therefore,  a  new  test  Is  needed  due 
to  the  Inadequacy  of  the  modified  Westenberg  tests  to  dl scorn  such  a  situation. 

In  order  to  determine  whether  a  pair  of  observations  may  have  values 
appreciably  larger,  or  smaller  than  the  other  observations,  the  probability  of 
having  both  members  of  that  pair  of  observations  be  among  the  largest,  or 
smallest  In  the  combined  sample  should  be  Investigated.  This  Is  accomplished 
here  In  the  Length  of  Initial  Run  (LIR)  Test.  In  this  test,  the  larger  value 
In  each  pair  of  observations  Is  labeled  "A,"  while  the  smaller  of  the  observe-' 
tlons  from  that  source  Is  labeled  "B."  If  no  pair  of  observations  Is  drawn 
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from  an  underlying  population  considerably  different  from  the  others  (In 
particular  If  no  pair  of  observations  Is  drawn  from  an  underlying  population 
whose  location  Is  considerably  larger,  or  smaller  than  the  others)  then  the 
length  of  the  run  of  A's  In  the  combined  sample  before  the  first  B  and  the 
length  of  the  run  of  B's  before  the  first  A  should  not  be  too  short.  (Note 
that  run  lengths  can  vary  from  one  to  n.) 

II.  Mathematical  Theory  of  UR  Test. 

Under  the  null  hypothesis,  that  all  pairs  of  observations  were  drawn 
from  the  same  or  Identical  populations,  the  length  of  the  Initial  run  of  A's  Is 
Identically  distributed  as  the  length  of  the  Initial  run  of  B's.  This  discus¬ 
sion  will  only  be  couched  In  terms  of  the  Initial  run  of  A's. 

The  probability  of  having  an  "A"  as  the  largest  value  In  the  combined 
sample  Is  unity.  The  probability  of  having  an  "A"  In  the  second  largest 
position,  under  HQ,  Is  the  number  of  observations  not  Included  In  the  sample 

pair  that  the  first  A  came  from  (2n-2),  divided  by  the  total  remaining  number 
of  observations  available  (2n-l).  The  probability  of  having  a  third  A  In  a  row 
Is  the  previous  probability  multiplied  by  the  number  of  observations  not 
Included  In  either  sample  pair  that  the  first  two  A's  came  from  (2n-4),  divided 
by  the  total  remaining  number  of  observations  available  (2n-2).  Therefore,  the 
probability  (under  H0)  of  having  at  least  r  of  the  A's  before  the  first.  Bis: 

/  2n-2  x  /  2n-4  x  ,  2n-6  x  ,  2n-2r+2  x 

(  -  )  (  -  )  (  -  )•••*(  -  ) 

2n-l  2r-2  2n-3  2n-r+l 

If  exactly  r  of  the  A's  proceed  the  first  B,  the  probability  of  this  occurrence 
(under  HQ)  Is  the  above  expression  multiplied  by  the  probability  that  the  r+lth 

largest  value  Is  a  B.  This  would  be  the  number  of  B  values  whose  corresponding 
A  value  Is  among  the  members  of  the  Initial  run  (r),  divided  by  the  total 
remaining  number  of  observations  (2n-r).  Simplifying,  therefore,  the  probabil¬ 
ity  (under  HQ)  of  having  an  Initial  run  of  length  r  Is: 

C2n:21!.!12n-,r-,l)!r  whefe  k,,  s  Wk-2)(k-4>  .... 

(2n-2r)! !(2n-l)!  (stopping  at  2  If  k  Is  even,  or  1  If  k  Is  odd). 

This  further  simplifies  to 

(n-l)!2frl  (2n-r-l)ir  a  fn-ll!2r“1  (2n-r-l)!r 

(n-r)!2n"r  (2n-l)!  (n-r)!  (2n-l)! 

In  general,  If  N  is  the  number  of  observations  per  sample  and  n  Is  the  number 
of  samples,  then  the  probability  of  an  Initial  run  of  length  r  Is: 

(n-1) !Nr"*  (Nn-r-1) i(N-l)r 

(n-r)!  (Nn-1) ! 


Simulations  could  be  used  for  alternative  hypotheses  and  for  Irregular  numbers 
of  observations.  This  paper,  however,  Is  concerned  with  N*2  observations  per 

sample. 

Simulations  (see  Appendix  III)  were  used  to  determine  the  relative 
frequency  distribution  of  Initial  run  lengths  under  various  alternative  hypo¬ 
theses.  Each  alternative  studied  assumed  one  pair  of  observations  to  be  taken 
from  one  underlying  population  and  all  others  taken  from  a  second  underlying 
population,  with  a  few  exceptions  for  sensitivity  study  purposes.  The  accuracy 
of  the  simulations  was  examined  In  several  manners.  First,  both  underlying 
populrtlons  were  set  Identical  and  the  results  compared  to  the  frequency 
distribution  for  the  null  hypothesis.  Agreement  here  demonstrated  that  the 
closed  form  solution  for  the  null  distribution  Is  correct  and  also  that  the 
simulation  was  accurate  to  approximately  three  significant  digits  using  20,000 
replications  for  the  cases  of  Interest  shown  In  Appendix  II,  also  under  HQ. 

However,  under  any  alternative  hypothesis,  accuracy  of  the  simulation  Is 
degraded  due  to  the  fact  that  the  distributional  forms  which  the  pairs  of 
observations  are  being  drawn  from  are  not  exactly  what  they  have  been  repre¬ 
sented  to  be.  Table  I,  however,  provides  a  set  of  chi-square  "poorness"  of  fit 
tests  which  show  that,  In  the  case  Investigated  there,  the  distribution  Is 
almost  exactly  as  was  represented.  (Similar  results  were  obtained  using  other 
distributions.)  Table  Tl  Is  used  to  demonstrate  the  small  differences  In 
resulting  output  when  Inputs  are  varied  to  degrees  that  were  found  unlikely  to 
actually  occur.  (Note  that  the  differences  found  In  Table  II  were  of  only 
approximately  the  same  magnitude  as  In  Table  I.)  From  this,  It  Is  generally 
concluded  that  only  two  significant  digits  should  be  used  from  the  relative  and 
cumulative  relative  frequency  outputs. 

In  addition  to  the  type  of  validation  shown  above,  the  simulation 
results  were  compared  to  a  closed  form  solution  for  the  probability  of  a  run  of 
length  one  when  one  pair  of  observations  is  drawn  from  one  distribution  and  all 
others  from  a  second  distribution.  In  order  for  there  to  be  a  run  of  length 
one,  both  observations  from  the  same  pair  must  be  the  two  largest  (or  smallest) 
observations  In  the  combined  sample.  Therefore,  If  p  Is  the  probability  of  a 
run  of  length  one,  and  we  are  Investigating  the  Initial  run  of  A's,  and  only 
one  pair  of  observations  is  drawn  from  one  distribution  with  all  others  drawn 
from  a  second  distribution. 
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TABLE  la 


i 

\ 

■*  I 


Random  Nos. 


True  Generated  From  A 


1 

- 

N(12,l) 

VS 

N(12,l) 

I  CELL  # 

(HOBS 

CELL# 

#OBS 

CELL# 

#OBS 

CELL# 

#OBS 

■  1 

370 

26 

410 

51 

417 

76 

405 

2 

375 

27 

428 

52 

402 

77 

403 

3 

402 

28 

382 

53 

398 

78 

405 

!  4 

369 

29 

413 

54 

381 

79 

360 

5 

412 

30 

386 

55 

447 

80 

361 

:  6 

376 

31 

400 

56 

385 

81 

393 

1  7 

407 

32 

396 

57 

393 

82 

388 

\  8 

424 

33 

426 

58 

.  375 

83 

408 

9 

413 

34 

386 

59 

404 

84 

381 

j  10 

381 

35 

433 

60 

401 

85 

388 

3  11 

377 

36 

380 

61 

419 

86 

404 

V  12 

380 

37 

407 

62 

371 

87 

388 

1  13 

394 

38 

418 

63 

414 

88 

408 

l  14 

381 

39 

393 

64 

400 

89 

421 

E  15 

382 

40 

407 

65 

415 

90 

411 

h  16 

364 

41 

370 

66 

396 

91 

422 

E  17 

388 

42 

428 

67 

415 

92 

397 

fa  18 

398 

43 

411 

68 

429 

93 

395 

1  19 

423 

44 

395 

69 

389 

94 

398 

[  20 

427 

45 

405 

70 

424 

95 

394 

21 

381 

46 

347 

71 

417 

96 

414 

J  22 

398 

47 

413 

72 

406 

97 

429 

23 

393 

48 

445 

73 

439 

98 

392 

24 

394 

49 

415 

74 

404 

99 

420 

1  25 

370 

50 

392 

75 

392 

100 

417 

THE  CHI-SQUARE  VALUE  FOR  A  N(12,l) 
USING  40000  GENERATED  RANDOM  NUMBERS 
TESTED  AGAINST  A  N(12,l) 

IS  94.56 
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TABLE  lb 


Random  Nos. 

True  Generated  From  A 

N(12,l)  VS  N( 12, 0.95) 


CELL  # 

(HOBS 

CELL# 

#OBS 

CELL# 

#OBS 

CELL# 

#OBS 

1 

282 

26 

385 

51 

449 

76 

437 

2 

295 

27 

426 

52 

415 

77 

390 

3 

317 

28 

430 

53 

418 

78 

349 

4 

344 

29 

403 

54 

402 

79 

387 

5 

343 

30 

424 

55 

468 

80 

410 

6 

372 

31 

409 

56 

395 

81 

394 

7 

361 

32 

424 

57 

426 

82 

399 

8 

380 

33 

424 

58 

385 

83 

393 

9 

390 

34 

409 

59 

435 

84 

392 

10 

402 

35 

433 

60 

417 

85 

389 

11 

378 

36 

417 

61 

444 

86 

401 

12 

362 

37 

412 

62 

387 

87 

409 

13 

370 

38 

445 

63 

413 

88 

406 

14 

396 

39 

424 

64 

454 

89 

394 

15 

382 

40 

403 

65 

399 

90 

414 

16 

383 

41 

426 

66 

439 

91 

375 

17 

362 

42 

401 

67 

435 

92 

368 

18 

369 

43 

458 

68 

416 

93 

382 

19 

412 

44 

399 

69 

432 

94 

366 

20 

436 

45 

432 

70 

441 

95 

383 

21 

438 

46 

387 

71 

424 

96 

366 

22 

385 

47 

418 

72 

451 

97 

367 

23 

388 

48 

460 

73 

418 

98 

335 

24 

416 

49 

442 

74 

398 

99 

354 

25 

396 

50 

415 

75 

414 

100 

300 

THE  CHI-SQUARE  VALUE  FOR  A  N(12,0.95) 
USING  40000  GENERATED  RANDOM  NUMBERS 
TESTED  AGAINST  A  N(12,l) 

IS  302.45 
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TABLE  Ic 


Random  Nos. 

TrU(  Generated  Prom  A 

N(  12 , 1)  VS  N( 12, 1.05) 


CELL  # 

#OBS 

CELL# 

(HOBS 

CELL# 

#OBS 

CELL# 

#OBS 

1 

495 

26 

418 

51 

402 

76 

393 

2 

443 

2/ 

366 

52 

371 

77 

384 

3 

448 

28 

407 

53 

388 

78 

398 

4 

445 

29 

378 

54 

372 

79 

398 

5 

412 

30 

386 

55 

398 

80 

359 

6 

442 

31 

367 

56 

396 

81 

342 

7 

444 

32 

427 

57 

366 

82 

385 

8 

428 

33 

356 

58 

372 

83 

401 

9 

399 

34 

423 

59 

370 

84 

392 

10 

395 

35 

352 

60 

383 

85 

380 

11 

396 

36 

386 

61 

387 

86 

398 

12 

399 

37 

423 

62 

400 

87 

396 

13 

378 

38 

385 

63 

358 

88 

402 

14 

384 

39 

162 

64 

368 

89 

422 

15 

354 

40 

385 

65 

422 

90 

417 

16 

404 

41 

365 

66 

385 

91 

433 

17 

411 

42 

403 

67 

376 

92 

433 

18 

411 

43 

394 

68 

421 

93 

418 

19 

413 

44 

387 

69 

381 

94 

416 

20 

383 

45 

360 

70 

382 

95 

427 

21 

373 

46 

345 

71 

404 

96 

424 

22 

392 

47 

392 

72 

411 

97 

473 

23 

392 

48 

427 

73 

392 

98 

479 

24 

368 

49 

391 

74 

431 

99 

472 

25 

391 

50 

380 

75 

391 

100 

566 

THE  CHI-SQUARE  VALUE  FOR  A  M( 12 ,1.05) 
USING  40000  GENERATED  RANDOM  NUMBERS 
TESTED  AGAINST  A  N(12,l) 

IS  277.28 


TABLE  II« 


Input  distributions  art  N( 12.00,  1.00)  and  N(1Q.00,  1.00).  Number  of  samples 
from  each  distribution  Is  1  and  9  respectively.  The  random  number  seed  for 
this  run  is  6S5S7. 


test  for  the  Length  of  the  Initial  Run  of  A's  before  the  first  B. 

i 

Number  of  reel  leaden**.  20000 


Length 

Observed 

Relative 

Cumulative 

of  Run: 

Frequency: 

7275 

Frequency: 

Frequency: 

1 

0.363750 

0.363750 

2 

4391 

0.219550 

0.583300 

3 

2980 

0.149000 

0.732300 

4 

2123 

0.106150 

0.838450 

5 

1534 

0.076700 

0.915150 

6 

857 

0.042850 

0.958000 

7 

517 

0.025850 

0.983850 

8 

246 

0.012300 

0.996150 

9 

67 

0.003350 

0.999500 

10 

10 

0.000500 

1.000000 

TABLE  lib 


Input  distributions  art  N(12.00,  0.95)  and  N(10.00,  1.00).  Number  of  samples 
from  each  distribution  Is  1  and  9  respectively.  The  random  number  seed  for 
this  run  Is  65657. 

Test  for  the  Length  of  the  Initial  Run  of  A's  before  the  first  6. 

Number  of  replication*: 20000 


Lenath 

Observed 

Relative 

Cumulative 

of  Run: 

Frequency: 

Frequency: 

Frequency: 

1 

7406 

0.370300 

0.370300 

2 

4480 

0.224000 

0.594300 

3 

3018 

0.150900 

0.745200 

4 

2134 

0.106700 

0.851900 

5 

1419 

0.070950 

0.922850 

6 

806 

0.040300 

0.963150 

7 

469 

0.023450 

0.986600 

8 

203 

0.010150 

0.996750 

9 

55 

0.002750 

0.999500 

10 

10 

0.000500 

1.000000 
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TABLE  lie 


Input  distributions  arc  N( 12.00,  1.05)  and  N( 10. 00.  1.00).  Number  of  samples 
from  each  distribution  Is  1  and  9  respectively.  The  random  number  seed  for 
this  run  Is  65557. 

Test  for  the  Length  of  the  Initial  Run  of  A's  before  the  first  B. 

Number  of  replications.:  20000 


Length 

Observed 

Relative 

Cumulative 

of  Run: 

Frequency: 

Frequency: 

Frequency: 

1 

7141 

0.357050 

0.357050 

2 

4273 

0.213650 

0.570700 

3 

2969 

0.148450 

0.719150 

4 

2119 

0.105950 

0.825100 

5 

1611 

0.080550 

0.905650 

6 

958 

0.047900 

0.953550 

7 

556 

0.027800 

0.981350 

8 

284 

0. 014200 

0.995550 

9 

78 

0.003900 

0.999450 

10 

11 

0.000550 

1.000000 

where  gj  2  is  the  d1str1but1on  the  k  “  order  statistic  out  of  2 

observations  In  distribution  1  (distribution  1  Is,  In  general,  the  distribution 
of  larger  location  If  the  Initial  run  of  A's  Is  being.  Investigated);  and  g2  ^ 

Is  the  distribution  of  the  k  —  order  statistic  out  of  m  for  distribution  2. 
Since  there  are  n-1  pairs  of  observations  taken  from  distribution  2,  m  =  2(n-l) 
«  2n-2,  and  therefore: 

1  1  1 
. . ■■ . —  "  ■  —r.—  -  ■  n  1 

m-1  2n-2-l  2n-3 

The  expression  for  p  Is  based  on  the  fact  that  If  both  observations  of 
a  given  pair  have  larger  values  than  any  other  observation  in  the  combined 
sample,  then  the  6  value  associated  with  the  largest  A  has  to  be  larger  than 
all  other  2n-2  values. 

Once  again,  use  of  this  validation  technique  supported  the  conclusion 
that  two  significant  digits  should  be  used  In  the  results. 

If  fj  (x)  Is  the  density  function  for  distribution  1,  and  f2  (x)  for 

distribution  2,  an  approximation  can  be  made  for  p  when  the  number  of  observa¬ 
tions  drawn  from  each  distribution  are  equal,  or  nearly  equal,  and  very  small. 
This  approximation  will  be  vary  poor  unless  the  assumptions  are  enforced.  In 
general,  however,  the  calculations  are  much  easier  than  those  In  the  previous 
expression.  In  this  case  the  approximation  Is  as  follows: 


♦  "  f  V*)  j  fl  (*)**•**  *  "  ♦/(!-♦) 

x  *  -«•  t  ■  x 


where  v  Is  the  number  of  observations  taken  from  the*  first  distribution. 

+  Is  the  probability  that  If  one  observation  were  drawn  from  each  of  the 
two  distributions,  the  observation  from  distribution  1  would  have  a  larger 
value.  The  approximation  Is  therefore  a  weighted  counting  procedure  which  doe* 
not  fully  account  for  the  shapes  of  the  true  distributions  of  Interest. 

The  tables  of  Appendix  II  provide  power  Information  for  a  variety  of 
cases  when  the  number  of  samples  (of  size  two  each)  Is  5,  10,  20,  and  60.  The 
alternative  hypotheses  oould  have  more  than  two  underlying  distributions  (up  to 
n)  but  two  are  sufficient  to  Illustrate  what  Is  being  Investigated  here.  The 
Importance  of  this  test  Is  to  determine  the  likelihood  of  having  one  (or 
possibly  more  than  one)  pair  of  observations  drawn  from  an  underlying  popula¬ 
tion  which  Is  substantially  different  from  the  underlying  population  from  which 
the  rest  of  the  observations  were  drawn. 
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If  sampl •  sizes  are  large,  It  Is  possible  to  reject  HQ  when  the  truth 

is  close  enough  to  HQ  for  practical  purposes,  unless  a  specific  Hj  Is  used  for 

a  power  analysis.  When  sample  sizes  are  small,  as  In  the  case  here,  one  may 
fall  to  reject  HQ  when  the  truth  Is  not  close  to  HQ.  Power  analyses  would  help 

by  completing  the  quantification  of  the  problem.  Without  a  power  analysis,  a 
hypothesis  test  Is  only  half  completed.  Null  and  alternative  hypotheses  work 
In  pairs  analogously  to  confidence  limits.  In  the  present  situation,  a  power 
analysis  Is  very  Important  due  to  the  unconventional  nature  of  the  problem. 

(It. Is  Interesting  to  note  that  when  the  number  of  samples  [of  size  two  each] 
Increases,  the  power  level  at  a  given  significance  level  remains  apparently 
approximately  constant  [see  Appendllx  II].) 

This  paper  makes  use  of  a  much  neglected  application  for  simulation. 
Simulation  can  be  used  as  a  check  for  a  closed  form  solution  when  the  develop* 
ment  of  such  a  solution  was  subtle,  In  addition  to  handling  situations  where  a 
closed  form  solution  Is  difficult  If  not  Impossible. 

III.  Example. 

Suppose  that  ten  processes  (or  Items  of  equipment,  etc.)  are  to  be 
examined  for  a  certain  trait  and  that  It  Is  expected  that  they  will  all  behave 
similarly  for  that  trait.  Also,  suppose  that  the  expense  Involved  In  studying 
those  processes  Is  great,  or  that  for  some  other  practical  reason,  the  number 
of  observations  per  process  (Item,  scenario,  etc.)  must  be  kept  extremely 
small.  If  two  observations  each  are  used,  a  combined  sample  size  of  20  Is 
obtained.  Whether  or  not  these,  samples  should  be  combined  would  then  be  open 
to  examination.  In  addition  to  any  subjective  arguments,  the  modified  Westen- 
berg  tests  and  the  LIR  test  should  be  applied  to  assist  In  this  examination. 
Suppose  (for  use  In  the  LIR  test)  that  the  larger  value  In  each  pair  of  obser¬ 
vations  Is  labelled  "A,"  and  the  smeller  values  labelled  "B."  Subscripts 
could  be  used  to  denote  which  process  Is  being  represented.  (This 
will  be  used  In  the  modified  Westenberg  tests.)  Suppose  that  when  the  values 
for  each  of  these  observations  are  ranked  from  largest  to  smallest,  the  follow¬ 
ing  result  Is  obtained: 

A8  B8  A7  A2  A1 j  A4  A5  A6  A10  A3  B2  A9  B7  B4  B1 1  B6  B10  B3  B9  B5 

The  number  of  times  thet  «  pair  of  observations  are  both  found  on  the  same  side 
of  the  median,  NZ,  Is  2.  Also,  the  number  of  times  that  a  pair  of  observations 
are  both  found  either  Inside  or  outside  the  Interquartile  range  Is  NZ«2.  Table 
III  shows  results  teken  from  the  progrim  of  Appendix  I.  PA  Is  the  probability 
level  of  the  test  associated  with  HQ,  and  PB  Is  the  probability  level  associ¬ 
ated  with  H^,  An  examination  of  this  table  shows  that  there  Is  no  good  reason, 

based  on  this  ranked  data  and  aside  from  subjective  arguments,  to  conclude  that 
these  samples  should  not  be  combined,  but  actually  then  Is  good  reason  to 
conclude  thet  such  a  combination  Is  advisable.  However,  the  LIR  test  can  be 
used  to  show  that  even  though  general  compatibility  appears  evident,  the 


TABLE  III 


HUM  MC 

HO:  *ACH  SAMPLE  HAS  AT  LEAST  lOOxRAX  CHANCE  OF  HAVING 
ONC  OBSERVATION  INSIDE  AND  ONE  OBSERVATION  OUTSIDE 
THE  INTERQUARTILE  RANGE. 

HIS  KACH  SAMPLE  HAS  AT  l.&ABT  lOOxRBX  CHANCE  OF  HAVING 
BOTH  OBSERVATIONS  FALL  TOGETHER 
ESTHER  INSIDE  OR  OUTRIDE 


inputs  are: 

NS#  THE  NUMBER  OF  BAMPLEJ 
NL»  -'ME  NUMBER  OF  ZEROES 
RA  AND  RB 

THE  NUMBER  OF  ZEROES  IB  THE  NUMBER  Or'  SAMPLES  HH06E  THQ  OBSERVATIONS 
ARE  FOUND  TQQETHER 


INTER  NBrNZrRAtRBs 
10  >  2  r  0 • 89  *0.30 
PA«0.2ZSZ  PB-0.\1S7S 
Do  you  wloh  to  run  tno  toot  aoalnT 
Entor  “Y"  or  "N*. 

Y 


ENTER'  NB.rNZrRA.RB-. 


Initial  run  of  A's  Is  extremely  short  (one)  and  this  Indicates  that  sample  8 
may  provide  values  too  large  In  comparison  to  the  other  values.  The  LIR  test 
shows  that  if  the  null  hypothesis  (that  all  pairs  of  observations  were  drawn 
from  the  same  underlying  distribution)  It  true,  the  probability  of  an  Initial 
run  of  length  one  Is  0.053  when  the  number  of  samples  (n)  Is  10.  This  Is  the 
probability  of  a  Type  I  error.  However,  If  the  null  hypothesis  Is  "accepted" 
here,  the  probability  of  a  Type  II  error  (for  any  alternative)  Is  unity  since 
no  amount  of  evidence  against  HQ  would  then  suffice,  if  the  values  of  the 

observations  here  suggest  that  graph  A  Is  tho  result  of  a  reasonable  alterna¬ 
tive  nypothesls,  than  the  power  against  that  alternative  is  of  Interest  and  Is 
approximately  0.5$.  Against  the  alternative  used  to  arrive  at  graph  B,  the 
power  Is  0.36,  and  the  power  from  graph  C  would  be  0.12.  Note,  however,  that 
tho  alternative  shown  here  against  which  the  power  Is  lowest  Is  for  a  situation 
better  Investigated  by  the  t«sts  of  Appendix  I,  program  WC,  originally  found  In 
reference  2.  Note  also  that  even  If  all  20  Observations  here  came  from  the 
same  distribution,  that  Is  a  small  sample  to  use  to  determine  the  form  of  that 
distribution.  Therefore,  other  alternatives  Involving  distributional  forms 
other  than  normality  may  be  needed  to  complete  the  analysis. 

Suppose  Bg  and  Ay  were  not  as  shown,  but  exchanged  In  position.  The  values 

of  NZ  would  remain  the  same,  but  the  length  of  the  Initial  run  of  A's  would  now 
bo  two.  If  Hq  Is  rejected  when  the  observed  run  length  Is  2,  the  probability 

of  a  Typt  I  arror  Is  0.158.  Against  tha  first,  second,  and  third  altarnatlves 
mentioned  above,  the  probability  of  a  type  II  arror  would  be  0,44,  0.64,  and 
0.88,  respectively,  and  tha  power  would  be  0.76,  0.58,  and  0.34,  el  so 
respectively. 

If  a  run  of  length  3  Is  considered,  these  figures  are,  respective  to  the 
order  given  above,  0.307,  0,24,  0.42,  0,65,  0.87,  0.73,  and  0.62. 

Considering  the  above,  when  n*10  It  could  bo  deemed  reasonable  to 
reject  HQ  when  the  run  length  Is  2  or  leu*,  and  "accept"  It  whan  It  Is  3  or 

more. 

In  conclusion,  If  some  of  the  observations  In  the  example  given  earlier 
are  shifted  In  rank  It  may  affect  on*  or  more  cr  perhaps  none  of  these  tests. 
Also,  If  the  modified  Westenberg  tests  greatly  discourage  the  combining  of  the 
samples,  then  the  LIR  test  will  probably  not  be  very  useful.  In  using  the' 
modified  Westenberg  tests,  If  It  Is  not  possible  to  divide  the  observations 
into  groups  of  aqual  sir, a  (above  and  below  the  median  and  Inside  and  outside  of 
the  Interquartile  range),  tihen  apply  tha  tests  shifting  the  observ«.t1on(s) 
which  are  In  question  from  one  possible  grouping  to  the  other  and  average  the 
results  obtained.  Finally,  bacausa  the  Bernoulli  trials  In  tha  modlfltd 
Wastanberg  tasts  usad  hare  are  not  truly  Independent,  these  tests  are 
approximate.  Also,  some  RA  and  RB  values  may  be  Inappropriate.  For  example, 
in  Table  III,  RA  could  never  be  C .95;  however,  It  Is  used  to  Illustrate  the 
strong  conclusiveness  of  these  particular  results.  A  study  of  the  null 
hypothesis  Indicates  that  for  RA  *  0.5,  the  binomial  distribution  used  for  this 


3S 


test1  h«s  thicker  "tails"  than  warranted, 
described  by 


ti£!/2^V 

(2»S)!/2"s 


KS!/(NS-NZ)i  ,  s 
C(NZ/2)!32 


(The  exact  null  distribution  is 


Also,  the  true  null  distribution  Is  skewed  to  the  right,  but  not  appreciably 
when  NS  exenods  50.  For  smaller  numbers  of  samples,  WB  1  from  reference  2  Is 
of  usti  The  advantages  In  using  program  WC  are  that  understandable 
alternative  hypotheses  can  be  shown  for  decision  making,  and  a  large  number  of 
sample  pairs  can  be  handled  with  very  little  computer  time.  All  that  Is  really 
needed  Is  a  table  of  the  cumulative  binomial  distribution.  This  test  should  be 
considered  "quick  and  dirty"  as  a  preliminary  to  the  Implementation  of  the  LIR 
test.  The  LIR  test  Is  an  exact  test  and  Is  easily  and  meaningfully  applied. 
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APPENDIX  I 


FORTRAN  CODE  FOR  Modi f ltd  Wtstenberg  Ttst 
(Deslgntd  for  n  pain  of  observations) 

This  program  was  rtferrod  to  as  "WC"  In  reference  2.  Not*  that  unllkt  tht 
other  modified  Westenberg  tests  of  reference  2,  which  are  exact,  this  one  Is  an 
approximate  tost. 
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1 

«► 

i 
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iioo 

/ 

/ 

/ 

/ 

/ 

./ 

/ 

/ 

■  /. 

1000 

999 


1000 

2001 

2002 


200) 

9010 


CHAKACTCMM  ANSWER 

.*RlTfj(0 ,  1100) 

. FORMAT(SX> #hOl  EACH  JAMPLE  HAS  AT  LEAST  lOOxRAI  CHANCE', 

Of  HAVING', /,9Xr'0NE  OBSERVATION  INSIDE  AND  ONE' 

'  OBSERVATION  OUTSIDE' ,/  ,  VX,  '  THE  I.iTERuUAKTILE  RANGE.', 
//,SX,'HlJ  EACH  SAMPLE  HAS- AT  LEAST  100xRB%  CHANCE', 

'  01*  HAVING',/, 9X, 'BOTH  OBSERVATIONS  TALL  TOGETHER'  , 

/# 9X , 'EITHER  INSIDE  OR  OUTSIDE' ,//////, 5X , 'INPUTS  AKEi 
.  /,9X,'MS,  THE  NUMREp  0 T  SAMPLES',/, 

9X,'NZ,  THE  NUMBER  OF  ZEROES', 

/,5X,'RA  AND  RB',//,5X, 'THE  NUMBER  OF  ZEROES  IS  THE  NUMBER  OF' 
«HITr(S£99«H0SE  TW°  0aSERVAir0Ms'»/»5'»#ARE  FOUND  TOGETHER') 

FORMAT (//,5X, 'ENTER  MS,NZ,RA,RB'> 

READ  (  5 ,  A )  -'•S ,  N i ,  n  A ,  KB 

lBNfi/2 

JbNZ/2 

PA-0 

P»«0 

MO 

X«1 

MK 

MO, 9*1 

ir(T,GI.M)GUTO  190 
P«X‘ 

w/(j>n 

.MO.. 

tfttl 

piP*(X«T), 

P«P/CX*¥-T) 

LoX*»y«T 

IF<L.GT,1)GS/T0  9 

.*»•<*,  G.T.**!,)GOTU.  0  . . . 

P0«PB4P*(<RBmm(l-kB)M(I«K))) 

,MK4l,  . 

XF<K,ie,J)GUIO  2 
K«K-1 

PA«PA4P*(  ( (  i-HA)**K)*(RAB*U-K)  )  ) 

_ ; _ 

IFCR-I)2,3,TOOO 

mx-y 

GOTO  ) 

*RITE(6*2O0 J )PA,P8 

FORMAT! IX, 'PA*',K6,4,5X, #PB»',Fb,4) 

WRXTC(6,2002)  ' 

F0RMATCT2 , 'Do  you  wish  to  run  th#  tost  agtln?',/, 

T2,'Entor  "2"  or  "N",') 

READ ( 9, 20U 3) ANSWER 
FORMAT (A) 

IF (ANSWER, EG, 'T' ) GOTO  1000 
STOP 


5 


APPENDIX  II 

POWER  TABLES  FOR  UR  TEST 


The  distributions  from  which  the  Input  Is  to  be  drawn  for  each  of  the 
alternative  hypotheses  are  shown  here  followed  by  histograms  of  the  relative 
frequency  distributions  for  these  alternatives.  In  all  cases  here,  both  of  the 
distributions  from  which  the  samples  are  drawn  are  of  the  same  type  but  with 
parameters  which  differ  In  seme  respect.  The  samples  could  have  been  drawn 
from  totally  different  distributional  forms,  and  more  than  two  such  distribu¬ 
tions  could  have  been  used  (up  to  n);  however,  what  Is  used  here  Is  sufficient 
to  demonstrate  this  test  under  conditions  which  Illustrate  Its  usefulness. 

In  the  case  of  the  normal  distribution,  when  standard  deviations  are  the 
same,  the  symmetric  nature  makes  the  relative  frequency  distribution  for  the 
Initial  run  of  A's  the  same  as  that  for  the  Initial  run  of  B's,  if  the  number 
of  pairs  of  observations  from  each  of  the  two  Input  distributions  Is  Inter¬ 
changed.  This  Is  true  In  all  cases  where  symmetric  Input  distributions  of 
equal  variance  are  used.  Also,  when  two  symmetric  Input  distributions  with  the 
same  location,  but  unequal  variances  are  used,  this  principle  applies.  When¬ 
ever  this  occurs,  the  output  relative  frequency  distributions  here  are  written 
In  terms  of  the  Initial  run  of  A's. 

The  Church-Harrls-Downton  (C-H-D)  method  of  testing  the  probability  of 
motor  case  rupture  In  missile  testing,  see  reference' 1,  makes  use  of  a  statis¬ 
tic  related  to  +  (shown  In  Section  II  here).  This  statistic  Is 

(A1!  *•  A^/C0*!2  4  <J“2 wh#ri  subscripts  "1"  and  "2"  refer  to  the  two 

Input  distributions.  For  any  of  the  tables  Involving  two  normal  distributions, 
If  /ij,  ai2.  a\,  and  are  changed  such  that  the  above  statistic  remains  con¬ 

stant,  then  the  output  relative  frequency  distributions  given  here  are  appli¬ 
cable. 

In  the  case  of  the  gamma  Input  distributions,  whenever  the  p's  (scale 
parameters)  are  both  multiplied  by  the  same  factor,  the  output  distributions 
are  still  applicable,  For  triangular  Input  distributions,  If  all  parameters 
are  added  to,  subtracted  from,  divided  or  multiplied  by  the  same  number,  the 
output  distributions  will  not  be  changed. 

The  tables  given  here  are  for  two  normal  distributions,  two  gamma  distri¬ 
butions,  two  triangular  distributions  and  finally,  two  beta  distributions.  The 
parameters  were  picked,  In  many  cases,  such  that  the  power  against  the  alter¬ 
natives  was  approximately  0.5  when  the  significance  level  was  approximately  0.1 
to  0.15.  This  occurs  for' H  ,  as  shown  In  the  following  table: 


#  of  samples 
of  size  2  ««ch 

S 

10 

20 

50 


run  length 
under  H 

o 

1 

2 

3 

4 

5 


(probability  level 
under  HQ) 

-11* 

-15% 

“15* 

“10* 

“15* 


The  following  tables  provide  a  variety  of  example!  of  alternative  hypothe- 
tes^and  result!  obtained  using  them.  It  Is  hoped  that  this  appendix  Is 
sufficient  to  provide  a  working  knowledge  of  the  power  of  this  test  to  Its 
Uiers.  When  any  specific  alternative  which  the  user  Is  Interested  In  Investi¬ 
gating  does  not  appear  here,  and  the  user  does  not  wish  to  spend  the  time  to 
get  the  programs  of  Appendix  III  to  run  at  a  convenient  facility,  It  Is  hoped 
that  the  results  can  be  Interpolated  from  results  provided  here.  Note  that 
following  each  graph  of  the  distributions  from  which  the  observations  are 
hypothesized  (Hj)  to  have  been  drawn,  only  relative  frequency  distributions  are 

provided.  The  hollow  bar  graphs  show  the  relative  frequency  distribution  under 
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Hgi  10  pairs  of  obiorvatloni  drown  from  tho  iimi  dWtrlbutlon 
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APPENDIX  III 
ALIRT 


As  stated  In  th«  Introduction  to  this  paper,  If  "A"  Is  the  larger  value  and 
"B“  Is  the  smaller  value  of  one  pair  of  observations,  then  the  probability  for 
the  Length  of  Initial  Run  (LIR)  of  "A"'s  or  the  probability  for  the  LIR  of 
"BM,s  can  be  used  to  Indicate  whether  data  has  been  drawn  from  the  same 
distribution  or  from  two  different  distributions. 

In  most  cases,  determining  the  probability  of  a  LIR  equal  to  N  (where 
N»l,2f3, . . . )  Is  Impractical  using  analytical  methods. 

The  purpose  of  this  prpgram  Is  to  use  simulation  to  estimate  the  probabil¬ 
ity  of  observing  a  LIR  equal  to  N  for  an  alternative  hypothesis  that  assumes 
data  has  been  drawn  from  two  different  distributions  rather  than  the  same 
distribution  (as  the  null  hypothesis  assumes). 


LIST  OF  VARIABLES 


Variables  usea  in  subprogram  initialize 

OiSTt  Value  , determined  by  input,  whicr.  specifies  the  probability 
distribution. 

lKH.Fi  input  value  tor  the  total  number  of  replications  to  be  per¬ 
formed. 

NKlLEi  Logical  unit  assignment  based  on  Input  value. 
name i  Name  of  output  file  (when  WF1LE«8), 

DiSPi  Disposition  of  output  tile  at  termination  of  program  (when 
NF  1LE«6 ) , 

ITEM*  Value  (A  or  B )  designating  the  type  of  run  test  to  do  performed 

LETTER i  Specifies  tne  type  of  probability  distribution;  ^-normal, 
(J-gamnia,  H-beta,  or  T-triangular . 

Variables  used  in  subprogram  SORT  j 

I 

sum t  The  total  number  ot  samples  drawn  (NSAmp(1)4-imsamp(2)). 

arum  Length  ot  tne  initial  run  ot  'A's  before  the  first  #b'. 

brum  Length  ot  tne  initial  run  of  'e's  before  the  first  *A', 

aresult (SUM )  i  Array  which  stores  the  number  ot  times  each  # A K U iv # 
lengtri  occurs  during  the  entire  simulation. 

BRESUiT (Sum )  t  Array  which  stores  the  numoer  of  times  eacn  ' b k u iv # 
lengtn  occurs  during  the  entire  simulation. 

Variables  used  in  subprogram  REPORT 

PERCENT (SUM) ;  Relative  frequency  destributlon  ot  tne  lir 
(ARESUL'l  (SUMI/IREP;  BRESULl  ( SUM ) / i REP) , 

CU m ( sum )  i  Cumulative  frequency  distribution  ot  tne  LIR. 

answer;  Input  value  which  determines  the  status  of  oar  graph 
output  of  frequency  distribution  (printing  or  Ml). 

Variables  used  in  subprogram  normal 


MU(I) i  Mean  for  distribution  l 


SIGMA (I)i  Standard  deviation  for  distribution  l. 

Xi  An  Independent  Identically  distributed  uniform 

random  number  generated  by  the  VAX  11/760  subprogram  'man'. 

T*  ( 2X“1 ) i  A  test  value  to  determine  acceptability  ot  the 
generated  random  numbers, 

Y*  N" to  # l ) . 

XI ,X2 I  M"(MU , SIGMA)  *s. 

Variables  used  In  subprogram  gamma 

ALPHA(I)  l-l ,2t  Shape  parameter. 

SCTA(Z)  i-i , 2 i  Scale  parameter, 

R*  An  Independent  Identically  distributed  uniform 

random  numoer  generated  oy  the  VAX  11/760  subprogram  'RAn#, 

test  6  wi  Test  values  used  to  determine  eccepteoliity 
of  the  generated  random  numbers, 

Yl  GAM"(ALPHA(l),l)  i«l,2. 

X  C 1 )  1-1,21  GAM" (ALPHA ( 1 ) , BETA ( I ) ) , 

Variable!  used  In  subprogram  bK'l A 

ALPHA ( I ,  J)  l,J-l,2i  Shapa  parameters. 

Ri  An  Independent  identically  dlstributad  uniform 

random  numbtr  genarated  oy  the  VAX  H/780  subprogram  #hak#, 

TEST  t,  at  Test  veiues  used  to  determine  acceptability 
of  the  generated  random  numbers, 

Yl  CAM" ( ALPHA ( I ) ,  1 )  1-1,2. 

XKI)  I«1,2J  GAM" ( ALPHA ( 1 )  ,  1 )  1-1,2. 

X2(l)  1-1, 2l  SET" ( ALPHA (X) ,  ALPHA (d) ) 

Variables  used  in  subprogram  thiang 
Gi  Minimum  value.  Location  parameter. 

Hi  Maximum  value,  Max-Min»5csle  parameter. 

Cl  Mode,  Shape  parameter. 

Ri  An  Independent  Identically  distributed  uniform 

random  number  generated  oy  the  v*x  11/760  subprogram  # h A « ' , 
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X2 ,x J  *  ThIANG**CO, l ,  (mode-min) / (max-min) ) . 

X ( 1 )  1*1,2*  TMlANG*'Cmin,max(mode) . 

Variables  common  to  subprograms 

NSAMP(I)  1*1,2*  Tne  number  o£  t wo-ooservation  samples 
required  tor  distrlDUtlon  l. 

A ( SUM )  SUm»i,n5a*P(1)+n8AMF(2) *  Array  ot  greater  values  ot 
each  two-observation  sample  from  the  combined  NSAMP'S, 

b(sum)  sum»i,nsamp(  i)  +  NSAMK2)  t  Array  of  lesser  values  ot 

each  two-observation  sample  trom  tne  combined  nsamP'S. 

count*  Counter  for  current  number  of  replications;  simula¬ 
tion  terminates  wnen  COUntbikeh, 

11*  initial  value  required  for  random  numoer  generation. 

PRUGKAM  LIMITATIONS 


Subprogram  tkianG*  The  Mode  *C'  used  to  generate  the 

TH1ANG (0,1,0,  where  C*(mode-min)/ (max-min ) ,  is  restricted 
to  u<c< 1  • 

Subprogram  GArfMA*  alpha>1. 

Subprogram  BET  A*  ALPhA  ( I. ,  J  )  >  1  1*1,2, 

input  distributions  must  both  be  of  the  same  type. 

input  distribution  type  is  limited  to  normal,  gamma,  beta, 
or  triangular. 

Tne  number  of  replications  must  be  less  tnan  or  equal  to  100,000, 
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qzE  nbaD— JfjT} 
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I  tffcPUKir.LAb/Tb 


r — re 

\  NO. 

XlMff  E 


#<snsED 

EH'^,B,Cn 


mr 

OF  SAMPLES  FOR 

C T PBT  nr^TRTm[TT| 


¥LLk.d'  r  ^,B,l,and 


NO.  OF  SAMPLES  FDR 

ME  SEOMLJULaiRIfliilll 
j? —  - 


■Hi 
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**********  ********M*t*MAIK  PROGRAM* »»♦*» ».»♦»»■**.»♦■»>.♦  JJ».ti.«Jl<»S>»a  *>.*«. 


COM MON /ALG/ A (50) ,B(50),SUM 

. CQMMON/ALL/NFILE,IREP,XIEiL - 

CQMMON/OUT/ARESULT(50) , BRESULK50) , PERCENT(SO) 

. COMMOW/FILE/RAMEf  DI3P _ _ _ _ _ 

CALL  INITIALlZE(DIST) 

. 15 . GOTUC1 ,2, 3, 4),DI3T _ 

1  ' CALL  NORMAL 

.  .  .  GOTO  10  .  ..  _ _ _ _ _ 

2  CALL  GAMMA 

.  .  00X0  IP  . . . . 

3  CALL  META 

...  coio  io . . . : _ : _ 

4  CALL  TR1ANG 

.10  .CALL  REPORT  . . . . . . 

STOP 

. END . . . . 


V*  ***********♦*"*****♦♦♦*•****♦****••***•*•••*••*••* 

*  Tni»  subroutine  Inputs  tho~type  of  probability  * 

*  ..  distribution  and  the  number  of  rablleatlons.  » 


SUBROUTINE  INITIALIZE (DIST) 

COMMON/AL.L/NTXHiIREPiXUM _ _ _ _ _ 

COM MON /FILE/ NAME fOZSP 

_  CHARACTER*!  LEXXEJUCHQ1CE _ _ _ 

CHARACTER*!!  NAME,DXSP 

.  U.  .  WRITE (6# 31 3 _ _ _ _ * _ 

31  F0RMATCT2, 'Output  mey  bo  written  to  e  file  and  then  printed 
/'or  printed  .at  this .tarsi  -Eater.  ?.f"  lor  file .nr  U1LZZ 

/  *  tor  terminal,') 

....  . BEAfl(5,.791ChJQICE _ ; _ - 

IF (CHOICE, EQ • *F') THEN 

. NFXLCwJ. . . . 1. _ _ 

ELSE  IF ( CHOICE, EQ, 'T' ) THEN 

. WFILEwb _ _ _ _ „.. _ _ 

GOTO  3 

. ELSE  ....  _ _ _ _ . _ 

GOTO  35 

.  END  It' . ' . . . — j _ 

WRITE! 6, 12) 

. 1 2  .  FO.RMAKxa#  'Sn t* r.  tbe.nAmt  llUi*). _ 

RSAD(5,17)NAME 

...  .1,7... .  i  ORAAKA) _ _ _ _ _ _ 

WRITE (6, 21) 

.21 . FORMAT CX2  # ! Select  the.  dltpeil  Uao  _o*-thQ  QUtPiit-illA«^4 _ _ 

/  /,T2, 'Typtl  KEEP, PRINT, or  PRINT/DELETE,') 

REAO(b,27)RI8P  . . . . 1 _ 

27  FORMAT(A) 

.  OPEN  t  S , F ILE«N AME, STATUS* 'NEW* #D1SP*DISPJ _ 

3  W H I T E ( 6  72) 

72  format (T2 » 'which  type  of  test  Is.  to  ht...psrioiuM4ti*/i.l2.» . . 

/  'Enter  " ARUN"  if  yog  wish  to  Know  tht  length  of  tho  A  ', 

/  'run  before  the  first  Enter  .".flJUlN "...  U  ^au.iLlin..tiui%.... 
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.  ,  .  -  ,  .  -  -  ’I  m  *  A  *’  '  l  *  •‘‘‘A  •"*  '  ** 


/  '  length  of  the  h  run  before  the  first  a.',/,T2,'Enter  "BOTH 

/  'it  you  woulu  like  the  result#  of  both  teste.') 

RfcADCb , 79JTE5T 
79  FORMAT  (A) 

IFCTESI.EQ.'ARUN'JTHEN 

iTEM*l 

ELSE  1FCTEST. EQ  .'bRUN')THEN 
.  ATEM*2 

ELSE  IF ( TEST  ■  £Q  •  'bUTH '  )  THEM 
ITEM*3 
ELSE 
GOTO  3 


End  if 

13 

wRITE(S,1U) 

10 

fur mat (12 , 'select  the  type 

of  distribution  you  would 

/' 

'like.',/,'  Type  the  first 

letter  of  the  name  to  metre  ', 

/ 

'your  input.') 

fit AO (b, 23) LETTER 

23 

FURhATC A) 

XF(LETTER.KU.'N')THEN 

01ST»1 

else  i  f  i  Letter  .  kg  ,  'g  ')  i  hf.n 

0IST«2 

ELSE  1 F  (  L  ft  TTEH  .  fcU  .  '  a  ' )  f  H  E  H 

oiai»3 

ELSE  1F(  LETT  EH  .  EG  »  'T  '  )  XHfc'N 
D1ST«4 
ELSE 
GOTO  13 
Erin  IF 

92  4HnK(6#93) 

93  FORM at ( T2 , 'Enter  the  number  of  replication!  ', 
/  'to  be  performed.') 

91  READ  ( 5  « *  )  IREP 

IFUREP.GT.lOgOOOJGOTQ  92 

RETURN 

END 


****************************************************************** 


*  * 

4  This  subroutine  generates  random  numbers  * 

*  from  tna  normal  distribution.  ♦ 

*  * 


************************* *****f**************************** ******* 

SUBROUTINE  NORMAL 

DIMENSION  V(2) ,MU(?),SJGMA(2),NSAMK2) 
CUMd0N/ALG/A(5*O),H(b0),SUM 
COMMOR/ALL/NF1LE, IHEP, ITEM 
CO'*MQN/FILK/NAi4E,  D1SP 
CHARACTERS  NAME, LISP 
PEAL  MU 

INTEGER  COUNT, 11, SUM 
COU NT* 0 

IF ( NFILK.  EQ.fOGGTO  fa 

CjPENttf,FlLE*NArtE#OTATUd*'OLO',DiSP*DISP) 

PRINT  5 

FOR m A T{ IX, 'Enter  mu,  sigme  and  number  of  samples', 

/  '  tor  tne',/,T2, 

/  'first  normai  distribution  separated  oy  comma*,') 
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fa 

b 


1 


Ri’AD(5 ,  * )MU  ( 1 ) , flIGMA (  1 ) ,  N3AMP (  i  ) 

WRI1E(6#17) 

FQRMAI.CT2#  'Entar  mu#.  sign*  and  numbtr  e<  samples', 

/  '  i or  tha',/,T2, 

- I — .'.second.  normal  distribution  separated  by  commas.') 

READ(S,*)NU(2),SIGMA(2),NSAMP(2) 

. . NRIXEifi#19.) . 

19  roRMAT(T2# 'Enter  a  five-digit  number  tor  random', 

_ L . '_n.umb.ir.fisriaration»') . 

R£AO(9,«)Xl 

_ RMTIXNf  2lI,9S)HUCi )  ,SXGNA(  1 }  ,NU(2)  ,SIGNA(2)  ,NSANPC1) , 

/  NflAHPC),Il 

.  JI.  ..FQRNATda# 'Input  distributions  or*  (r<'#r9.2#'#'#r9.2, ')  and', 
/  '  N"('#F3(2,*,'rB. 2, '),',/,T2, 'Number  of  samples  iron  tech  ', 

_ i .. '^litrlbutlon.  1 1.  .* « 12#.'.  and  ...' ,  . 

/  22#'  respectively, ',/,T2, 'Tho  random  numbor  food  for  this  '# 

_ IS# '  # ' J 

97  COUNlmCOUNT^l 

_ jlffl . . . . . 

SUMO 

— JLQQ _ . . . . 

N«0 

_ M _ I.Sfi _ _ - . 

DO  10  I»l,2 

_ ItR/MLtIU.. . 

VU)-2.*X-1. 


10 

TaTtM 

..fifliZXflIUL. . . 

ir(T,GT.l,)GO  TO 

.  YlPLOfltT) . . 

Y2e-2.*Y1/T 

.  ...... 

'  lAlOltTiYl.) - - 

N«Ntl 

— JULHslJlNll . . 

xiMU(J)4«lGHA(j)av(ijay 
— X211UUU2  4AlflMAWl*YC  2 ) « 
IF(X1 ,GT«X2)THEN 

...AliUHlKli . . . . . . 

B<BUH)mX2 

— KLAC. . . . -  . . . 

A(«UN)aX2 

.....  JIULJ.M.).P.X1.  . . . 

ENDIF 

— l££ii#LT«N8AHPt  J)  )  GO .  TO  90 
Zr(G.bT.2)GO  TO  100 

.-L.OJ4I1  SORT . 

1F(C0UNT«LT, IREPJGOTO  97 

_ RETURN . „ . 

END 


l 

. . . 0 .  .  *. 

*  This  subroutine  generates  random  numbora  0 
-i- . . . A .  from.tno,  gemma  distribution.  .,  * 

0  0 


SUBROUTINE  GANNA  ■ 

QIMCNSION  U(2),Xm,AtPHA(2),BETAC2),NSAMP(2) 
98 


v*  -i’.  .*•  .■* .  - . 

v<.  .  “i  •.*  k 


•/  -.i' 


COMMOl'J/ALCi/AtSO)  ,H(SO),8UM 
COMrtQiM/ALL/HF  ILE  ,  XHEP  ,  IT  EM 
COMMON/ 1*  ILt/NAME , D18P 
CHAHACTEHM  J  NAMF.,DI3P 
J NTEGfcN  COUNT ,11 , SUM 
NEAL  UTHETA , LNZ 
COUNTbO 

l  F’  (  NFILt .  KU .  6  )Gf)TU  b 

OPEr.  C0,KILE*NAMfc;,  STATUS* 'OLD  '  ,D1SP*U1SP) 

6  PRINT  10 

10  F’OKmATUX, 'Knttr  alpna,  beta  and  number  ot  samplei  tor', 

/  '  tne',/,T2, 

/  'first  garma  distribution  separated  by  commas*') 
REAB(9,*)AU»HA(l),eKTACl),N8AHPU) 

IF (ALPHA l 1 ), Lfc, 1)0170  b 
e  PRINT  11 

11  FGHMATUX* 'Enter  alpha,  beta  and  number  ot  samples  .tor ' , 

/  '  tna',/,T2, 

/  'second  gamma  distribution  separated  by  commas.*) 
HEAi)(!i,|l|)ALPHA(2),hKTA(2),NSAMP(2) 

IK(ALPHA(2).Lt.l)G0T0  0 
PRINT  12 

12  FUKMATU2, 'Enter  a  five-digit  number  for  random', 

/  '  numoer  generation,') 

KEAD(b,*}ll 

toklTKCNflLK, 98) ALPHA! 1) , BETA C 1 ) , ALPHA! 2) , BETA ( 2) , NSAMP ( 1 )  , 

/  M4AMPC2) , II 

9b  FOH.-iAT(12, 'input  distributions  art  C*( ',F9.2, ' , ' ,l’5, 2, ' )  and', 
/  '  G**!', F5,2,','K  5*2, ')•',/, 12, 'NUmbar  of  samplas  from  ', 

/  'each  distribution  is  ',12,'  and  ', 

/  12,'  raapactlvaly.',/,T2,'Tht  random  numoar  stag  for  thla  ', 

/  'run  is  ',In,'.') 

THEIAaA.S 

l/J  HETA*LU(j  ( 4  .  b ) 

D"1 ,+LTHElA 
F‘l)lJrt*LOG  1 4  «  ) 

97  CUUuTaiGUUHTtl 
3  U  N*  ■  0 

k*o 

60  K ■ K ♦ 1 

Gi«2,*ALPHA(K)-l, 

G2*£»QHT(Gl) 

G*i./G2 

Q*AbPhA(,K)+G? 

H«ALPhA!K)-FULH 

N*0 

80  0*0 

20  JaJM 

50  I’U  JO  1*1,2 

P*RAN(11) 

0 ( I ) *H 

JO  CONfXMifc. 

vi*ua)/u,-uu)) 

V2*bt)G(Vl) 

V*f.  +  V2 
VHEXP(V) 

X “ALPHA ( K) *X1 
Z*U(2)*U(1)*UC1) 

TEA t*rt fb-lHETAeZ 
IFUEBT.GE.O.)^  Tl  4.i 

99 


LHZ«LUGU) 

1»'(M,CS.LNZ)G0  TCI  40 
GU  TO  50 

40  XCJ)aX*8ETA<K) 

IF ( J.LTt 2)G0  TO  20 

SUM«SUM+1 

N«N+1 

IF(X(i> ,LT.X(2))THEN 

A(8UM)*X(2)  . 

B(iUM)«XCl) 

ELSE  . 

A(8UM)aX( 1 ) 

B(*UM)«X(2) 

END  ir 

ir(N.LT.NSAMP(K))GO  TO,  8.0 
XF(A.LT.2)G0  TO  80 

CALL SORT  . 

IF{COUNT.LT.INIP)GOTO  97 

. RETURN  . . 

END 


*  ..  Thii  subroutlnt  gtnaratai.  rtnAOA.-MMfe>.M. _ 4..... 

*  from  tha  baca  distribution'  4 


) 

10 


11 


12 
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SUBROUTINE  BETA  . . . . . . - _ _ 

DIMENSION  Xl(2)iX2(2),U(2),ALi»HA<2,2},NSAMPC2) 

CGMRDN/ALG/A(5Q),8C50),8UM . : . . . 

COMMON/ A Lb/N FILE' I REP' ITEM 

COMMON/riLE/NAME'DISP. _  _ _ 

CHARACTER*!!  NAME,DI8P 

REAL.  LIHETA'LN*  .  . . . . . 

INTEGER  COUNT'll'N'O'SUM 

C0UNT40 . • . . . 

irCNFILE.KQ.filGOTO  3 

0PEN(l»FILE»NAM|'8TAT.US»/0LQf  iJ?I8PiPIflR) . . 

PRINT  10 


/ 

/ 


roRMATCiX' #Enttr  alphAl.i  alpha!  and.  numDar.Ql.  ..itiPlifl..-’.*. 


/ 

/ 


'for  th*''/'T2» 

'lint  btta  diatrlbutlon,  •ipa.rittd.by.  ee.niuJx') _ 

READC b, *) ALPHA (1,1), ALPHA (2, 1),N8AMP(1) 

PRINT  11  *  ...  . . . . . . 

FORMATC1X, 'Enttr  alphal,  alphas  and  numbtr  of  aamplaa 

'for  m',/,Ta«  .  . . . . . . 

'stcond  bata  distribution  aaparatad  by  commas* ') 

REAP  ( 8 , 4.)  ALPHA  C  1,2),  ALPHA  (2 ,2) ,  NSAMPtZ* _ _ 

PRINT, 12 

roPMT(T2,#Entar  a  flva-dlglt  numbtr. for  random.' ». . 

'  numbtr  generation. ') 

REAU(5'«m 


/ 

/ 

/ 


WRITE (NFXLE' 98) ALPHA (1 ' 1 ) , ALPHA( 2 , 1 ) , ALPHA ( 1,2),  ALPHA ( 2,2) , 

NSAMPC1),NSAMP(2)'I1 .  . . . . . - . . . 

lURwAKTS'* input  distributions  ara  B“(',F5.2, ',',F5.2,')  and', 


»"•(', rs. 2, ','F5. 2, '),',/,T2, 'Number  of  samples  from'. 


aaeh  distribution  Is  *,12''  and  ', 


12,*  respectively, ',/jJii 'Tha  random  numbtr  seed,  for  this 


100 


9  •  ■ 


/ 


'run  is  Mb,'.') 

THlif  A«4.& 
l’JHfcTA*LUGl4.5) 

H«1  .4liTHtl'A 
I1  OUR*JjOG(4  .  ) 

97  COUI.'TbCOUNTU 
D«0 
SU»«0 

DO  VO  iv ■  1 , 2 
Claim 

DO  <K)  K«l,lvSAMP(N3 

SUMaSDrttl 

DO  70  L»i,2 

DO  hO  Kal , 2 

G1«2.4ALPHACK,N)-1. 

GaaSwHKGl) 

Gal. /G2 

Ub  Al.PHA(K,N)fG2 
HBADPHA(K,N)-FOUH 
SO  DO  iO  lal,2 
RBhANUl) 

U(i)lK 

30  CUNlIivlIt 

Vlau(l)/(|.-UCl)) 

V2BLQG(V1) 

Vaf,»V2 

»1«6'XP(V) 

Y bAlPHA  (  K ,  tv )  *Y  1 

*aiuai*uii)*u<n 

l«BHt(ji*V-Y 
TL31'BW  +  rJ«THfcIA*Z 
irCmT.GK.'MGU  ICJ  40 
LN2aLOG(Z) 

If’fm.GK.LNajGO  TO  40 
GO  TO  50 
40  X 1 ( K ) a  Y 

60  CONTI  NUb, 

X2(b)*Xi(l)/(Xl(l)+Xl(2)) 

7o  cummin. 

IFCX2U  )  .GT.X2C2)  ITHKN 
A(fiUM)aX2U) 
R(5UM)aX2(2) 
ast 

ACOUI->)bX2(2) 

B  C9UM) »X2 ( 1 ) 

RND1F 

50  CUNTlinUlt 

90  CCII.  UN UL 

CALiU  60HT 

iFCCOURT.Ll.lKePJGUTO  97 
PBTOHff 
l  r. 


IffM*****'********************************************************** 


♦  4 

*  This  subroutine  generates  random  numbers  * 
e  from  the  triangular  distribution.  * 

4  * 

**t**M»«*****4***t****>MM**M  +  »M*iM**4*9*»******4t*t*****:M******* 


SUBROUTINE  TRIANG 

_ DIMENSION . mX»&m^2UC.<.21#ll&Altet2.J - - - - - 

COMMON /ALG/AC 50) , B  C  5 0  > ,  SUM 

.  COM MON/ ALL/NFXLE, 1REP, ITEM 

COMMON/ FILE/ NAME, DISP 
.  CHARACTERS^  NAME, DISP 

INTEGER  COUNT, II, SUM 

_ CQUNXaCL. _ _ _ _ . _ _ _ _ _ . _ _ _ 

IF(NFJLE,EG.6)G0T0  9 

_ OPEN (B ,FILEmNAME,STATU8*'0LD', DISPsUISP) 

9  PRINT  50 

50  _ FORMAT ( IX, 'Enter  minimum,  maximum,  mod*  and  number  o f  ', 

/  *  of  samples  from  the',/,T2, 

_ L — !±irjt triangular distribution  separated  by  commas,') 

READC5, *)G( 1 ) ,HC1),C(1),NSAMP(1) 

JJ_ _ PJIS3.  11 . . . 

51  FORMAT ( IX, 'Enter  minimum,  maximum,  mode  and  number  of  ', 

_ l _ !_At_aiJBplea  irom-the  '*/  ,12, . . . . . . 

/  'second  triangular  distribution  separated  by  commas,*) 

XFCGtl)  ,GT.H<i)  .OrTgU)  IgT.CU  )  •6ii«'C(T)7ov7H(i iT)TheV 

_ J1R1.TEC6, 3)  . . . . . 

7  FQHMAT(T2, 'INCORRECT  PARAMETERS  ON  DISTRIBUTION,', 

_ l _ * _ TRY.. AGAIN,  'J . 

GOTO  9 

- ILLIL.JLF^SX.2)  ,fiI,H(-2J  ,QR,G12)  .GT.C  C2).«0R,CC2)  ,GT.HC2) )  THEN 

WRITE(6,7) 

_ Aoxo.u. . — . - . . .  . . . 

ELSE 

_ PJUUI.  12 . . . . . . . . . . . . . 

52  F0RMAKT2, 'Enter  a  five-digit  number  tor  random  number', 

- 1 - L_atntr4Llo.ru '.2. - - - ...» . . . . . . . . 

READ(5,*)X1 

_ JENO..  IF  . . . . . . . .  . . 

WRITK(NFILE,98)G(1),H(1),C(1),G(2),H(2),C(2), N8AMP( 1 ) , 

_ A . N.3AKPX2)  ,  XI  . : . . . . 

91  roRMAT(T2, 'Input  distributions  are', 

_ A-.1JX1LUXA* tnd.%.  . . . . - . 

/  '  T“(',F5,2,',',F5,2,',',F5,2,').', 

— umber  .of.  samples,  .from  etch  distribution.  is.'* ... 

/  12,'  and  ',12,'  respectively,', 

_ /._Z,l2#*jrhi  ..r.andam . number  seed  tor  this  run  is  f,is,'.') 

97  CQUNT*COUNT+ 1 

_ jmpJL.... . . . _ . .  . . . . . 

00  30  I«i,2 

_ _ T.eHtD-GlI  1  . . 

CQMP«CCCI)-G(I))/T 

_ _ .£.o.mpi*i , -comp . 

00  20  L»1 , N8AMP ( I ) 

_ SUMAAIUIti . . .  . 

DO  10  d«l,2 

. _ . WSRAMXI) .  . 

IF ( U , LE , COMP) THEN 

. . . X1«.C0M.P*U 

X2«S0RT(X1) 

_ _ _ _XCJ)iGCI)-fT*X2 

ELSE 

. . . . XlM.CQMPl  e  ( 1  ,-U) 

X2*SQRT£X1 ) 

. . xa«i,-xa  _ 


x(j)*Gcn+r*x3 

ENDlF 

10  continue 

IF(X(1).GT.X(2))THEN 

ACSUM)sX(l) 

B(SUK)*X (2) 

ELSE 

A(8UM)*XC2) 

B (SUM) *X  C 1 ) 

ENDIF 

20  CONTINUE 
30  CONTINUE 
CALL  SORT 

IKCCOUMT.LT. IREJOGOTO  97 

RETURN 

END 


************************************************************************ 
*  * 

*  This  subroutine  performs  a  bubble  sort.  * 

♦  * 
************************************************************************ 


SUBROUTINE  SORT 
DIMENSION  AA(2).BB(2) 

CUMmON/ALG/A ( 50 ) .BOO)  .SUM 
COM MON/ALL/NKILE.IREP. ITEM 

CUMMC.VUUT/AKESULTOO).  B RESULT (50  ). PERCENT (50) 
CClHMON/ElLE/NAME.DISP 
CHARACTER* 1 3  NAME. DISH 
INTEGER  ARUN, BRUM. SUM 
12  INTERA* 1 

DO  1*1, (SUM-1) 

IF ( A ( 1 ) ,GE,A(1*1) ) JOT 0  25 
AA(2)«A(I) 

A  A ( 1 ) *A ( 1  + 1  } 

A ( 1 )*A A ( 1 ) 

A (141 )»AA(2) 

INTERA-O 
25  CONTINUE 
END  DO 

IF ( INTER A , £0.0) GOTO  12 
9  JNTERbal 

00  U*i,(SUM-i) 

1F(B(J).GE,B(J+1)) GOTO  27 
HR ( 2 ) «B ( J) 

BB ( 1 ) »B ( J*1 ) 

B(J)»BB(ii 
'B(J*1)*BB(2) 

I NTERH*0 
CONTINUE 
KND  DO 

IF(IIMTERB.EU.U)GOTU  9 
ARUN»0 
HRUrt»0 
00  1* i . SUM 

1 F  (  A  (  i  )  •  GT .  i'H  1 )  )  THEN 
ARUUxARUN+1 
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27 


ELSE 
GUTO  7 
END  IK 
END  DO 

7  CONTINUE 

DO  l»SUM,i,-l 
IFIB£I).LT»A(SUM) )THEN 
BRUN-BRUNM 

ELSE . 

GOTO  18 
END  IF 
END  DO 

18  CONTINUE 


ttaaaeaeeeea  ******  a  #  a  t  a  f j  f  _ 

*  End  of  replication  bookkeeping i  * 

*  This  part  of  the  subroutine  stores  J - - - - 

a  tna  statistic  Ne  of  runs  of  lanoth  * 

a  X"  in  the  array  RMULXtX) - A - . - 


DO  1*1 f BUM 

ir ( AKUN (E0. I )  ARESUtH  X)*ABEIULT(I1+A. 
IF ( BRUN • EO a  1 1 BRESULT ( I )*BRKBULT( I) al 

END  DO  .  ,  . . : _ 

RETURN 

END  .  .  _ _ _ _ _ _ 


LIXIXIXXXXIxXIIIXXlXirxrxriIIIIllXIIlXllXIllAXJ.AlA.tXlXAAAllI-lAXlAJAAJU.XAl 


a  This  subroutine . fl«nerem.J.tJi*_DJiUuiL_af - * - 

a  tna  simulation*  * 

*.. - -  - - - - * - 

tteaeaeaseaeaea* eseeeeeeeeeeeeeeeeeeeeeeeeeeeeeeeeeeeseeeeeeeaeeeeeeeeeei 


SUBROUTINE  REPORT  . . j _ 

DIMENSION  AA(2),BB(2),CUM<50),INT(50) 

CQMMQN/ALG/A  £50)  #.5£5Q)«.8UM - - ; 

COM MON/ ALL/NFI LE * IRBP, ITEM 

CQMMQN/QUT/ARESULTC50)  #  BRESULT  CSO)  *  PERCEN  T  ISJCL) _ 

COM MON /FILE/ NAME »DISP 

CHARACTERatl  NAME*DXSP#  ANSWER . . . . . . 

INTEGER  BUM 

IF  CNF1LE*EQ*6)G0TD  11 .. - - ■ - 

OPEN (8  rFXLE*NAME# STATUS* 'OLD' #DX8P*DXBP) 

IF  (ITEM* liQa  1  *OR*ITEM*EO.  J1THEN _ _ _ 

ASSIGN  37  TO  10UT 

ELSE . . . . . . 

ASSIGN  39  TO  XOUT 

END  IF  . . . . 

WRITE (NFILEjIQUT) 

FORMAT(///,T2* 'Test  for  the  Length  af  the  .Initial  JLua. 
'  A  "s  before  the  first  B,',/) 

format(///,T2, 'Test  for  the.  Length  of  the ...Xnl.tlil.Bun. 
'B"s  before  the  first  A,'*/) 

WRITE (NFILE* i))IREP  - - - - - - 

FORMATCT2, 'Number  of  Hepllcetionsi  ' , 1 6 , / 3 

IFC1TEM (EOi  1 « OR,  ITEM t EOa  3JTHEN  . ...._ . . 

WRITE(NFILE,42) 

form  AT  ( T2  >  'Length  'f12U  'Observed  _ _ _ _ 


/  ' 'Relative  #f4Xf 'Cumulative') 

mTE(NFILE,44) 

_ 44.  FQRMAT(T2i 'of  Run! ', 4X , 'Frequency i 4X, 

/  'Frequency! ',4X, 'Frequency! ') 

.  T»XREP 

CUM(0)»0.0 

.  DQ  lei, SUM 

PERCENTd )«ARESULTd)/T 

- .,CUMII)«CUMU-11+PERCCNT(I> 

END  00 

.  CO  I*1«9UM 

INT(I)«ARE5ULT(I) 

. .  . WRITECNFIUE,  17)Xi  INTd) » PERCENT  (I)  *CUM(I) 

17  rORMAT(I2,X3,lta,I6pT20,r8.6,T42,Fi,6) 

_ _ _  UNO .  DO  .  . 

MRlTECNFXLEr 62) 

.  62 . P0RMATCT2, /,/,/,/) 

CALL  GRAPH 

. .  .  lFCXTEM.EQ.DGOXO  57 

XTKMa2 

. .  GOXO  11 

ELSE 

_ _ «RITE(NFXLE#42) 

WR1TE(NFILE#44) 

. . . . TflRBP . 

CUMCO)»d,Q 

_ _ do.  im,auR. . . . 

PERCENT (X)«BRESULT(I)/T 

.  CUMd)»CUMd-l)+PERCENT(I) 

END  DO 

. .  00  X*1#SUM 

lNT(I)»BRESULt(l) 

_ _J~-,  WRITE (NFILE*  17}X«INT(I)r  PERCENTd) ,  CUM  Cl) 

END  DO 

. WRXXE(NF1LE«62) 

,  CALL  GRAPH 

. . END  IF.  .  .  .  . 

67  ’  CONTINUE 

_ :-.REIURN... . .  . . 

END 


..lkf.e*..f.iP.!M!fi**fM******e*»**M**4e********e**MfMM*MMMM$MM***e* 


*  » 

. .....* . ThU  subroutine  generates  the  relative  end  * 

*  cumulative  frequency  dlatributlon  * 

... .  .  function  graphs*  * 

*  * 


*****+******  *********************************************************** 


SUBROUTINE  GRAPH 

. DIMENSION  TEST 1 (50) *  CUM (50 ) 

COMmON/ALG/A(50),B(50),SUM 
COMMON/ ALL/NFX LE , XRBPfXTEM 

CQMMON/OUT/ ARE5ULT (50)|BRE5ULT(50)*  PERCENT (SO) 

_ _ .CQMMQN/FXbC/NAME,DX0P 

CHAKACTERM3  NAME ,  DISP  ,  ANSWER 
..  INTEGER  SUM 

IF(NFILE.EQ«6) GOTO  13 

...  .  OPEN(U, FILENAME, 5TAT£Sii^0L&',DXSP»DISP) 
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ii  IFU'fEM.EG^JTHEN 
ASSIGN  99  TO  Li 
ASSIGN  S8  TO  K 
ASSIGN  84  TO  M 


ELSE  . -  . 

ASSIGN  9B  TO  L 

ASSIGN  86  TO  K  . 

ASSIGN  82  TO  M 

END  IF  ...  . . . . . . 

wftITE(6#L) 

98  format (T2# 'For  this  ttst  of  the  Itngtn  of  the  run  of.'_# _ 

/  '  a"s  before  the  first  B,',/#T2, 

/  'do  you  wish  a  bar  grapn  of  the  relative  and'# . . 

/  '  cumulative  frequency  distrlbutlons?'#/#T2# 

/  'Enter  "YES*  or  "NO".')  _ _ 

99  formax(T2# 'For  this  test  of  the  length  of  the  run  of'# 

/  '  b"s  before  the  first  A#'#/#T2#  _ _ _ _ 

/  'do  you  wish  a  bar  graph  of  the  relative  and'# 

/  '  cumulative  frequency  distributions?'#/#!**  _ _ _ _ 

/  'Enter  "YES"  or  "NO".') 


HEAD (St  91) ANSWER  . 

91  FORMAT! A) 

If (ANSWER, EQ.'NO'IGQTO  89  . 

WKIU(NFILS,27) 

27  FORMAT  ('!')  . . 

WHITE (NF1LE#K) 

86  FORMAT (TS# 'Relative  frequency  graph'#  .  . . . 

/  '  for  tna  Length  of  the  Initial  Run  of'# 
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37 


31 

33 


/  '  A"e  before  the  first  8.'#/)  . .  . . . . 

FORMAKTS, 'Relative  frequency  graph', 

/  '  for  tna  Length  of  the  Initial  Run  of'# _ _ _ _ _ 

/  '  B"s  before  the  first  A.'#/) 

wRIIE(NFILC#  37)  _ _ _ _ 

FORMAT (T 5# *0«00'#T10# ',04'#T14# '»08',T18# *#12'#T22# '.16'# 
/Tib , ' .  2U '  # T30 , ' . 24 '  # T34 , ' .  28 ' , T38 , ' .  32 '  #.142# JA.'  JKui .IQ'*. 
/T30,'.44',T54,'.4B',TS8,'t52'#T62,'.56',T66,'.6C',T70#',64', 
/T74,'.6S'#T78#'.72',T82#'.76',T86,'.M'#T9Q.,'..M.'*M.4*'.*18f, 
/T98,'.92\T102#',9b',n06#'l,Q0') 

.  WRITE(NFILE#33)  ...  . . . . 

FORMAT  (16#  +  '  , 


/  # . 

/  '+•  +  •  +  •  +  »  +  >  +  »  +  •  +  + ■  +  •  +  •  +  •  +  •  +  ■  +  >  +  « +*4*4«>  +  «+  '  ) 


CUM(0)*0.0  . 

DO  1*1 1  SUM 

TE6T1  (1)«(  PERCENT  (i)t.OOS)  4 100Q. . . 

JaTESTKD/lO. 

IF  ( J .  EU.  0  )  THEN  . . 

ASSIGN  43  TO  IOUT 

ELSE  . 

ASSIGN  41  TO  10UT 

END  IF  . 

CUM(I>«CUM(I-1)4PERCENT(I) 

WRITE(nFILE#10UT)I,PERCENT(I) 

41  rOHMAT('Q',TJ,I2fT6,'-',<d>(iH+),2X,F8.J) 

43  rORMAT('0'#TJ,12#T6('-',2X,FI.J) 

END  DU 

WR1TECN FILE/27) 

WRITF(NFILE,M) 

R2  FORMAT (T5, 'Cumulative  frequency  graph'# 

/  '  tor  the  Length  ot  the  Initial  Run  of'# 

/  '  A"s  before  the  first  B . ' .  / ) 


106 


I 


I 


14  FORMAT (T5, 'Cumulative  frequency  graph', 

/  '  for  the  Length  of  tne  Initial  Run  of', 

..J  B"s  before  the  first  A,',/) 

WRITE (NF11E» 37) 

WRITE! NFILE, 33) 

DO  1*1, SUM 

TESTl(l)*(CUM(I)*.005)*1000. 

JeTCSTi  ( J)/iQ* 

. .  IF(d.tQ.O)THEN 

ASSIGN  43  TO  IOUT 

. ELSE 

ASSIGN  41  TO  IOUT 

. .  END  IF 

WRITC(NFILE,10UT)1,CUM(I) 

. END  DC 

WRITE CNFILE ,  27 ) 

..  89  CONTINUE 
RETURN 
END 


85 

i 

i 


*****  ******  ***********************************  ***l,‘**  ******  *  ********** 
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ADDENDUM 

After  listening  to  the  presentation  of  this  paper,  Dr.  W.  J.  Conover  of  Texas 
Tech,  commented  that  perhaps  a  rank  test  based  on  the  rank  sum  of  the  A's  or 
some  other  appropriate  measure  might  be  used  as  a  more  powerful  test  of  overall 
compatibility  among  the  samples  with  emphasis  on  shift  of  location.  In  view  of 
the  weakness  of  the  modified  Westenberg  test  for  the  median  which  Is  given 
here,  and  the  generally  high  power  of  rank  tests,  this  suggestion  seems 
promising.  I  would  still,  however,  suggest  the  LIR  test  for  the  purpose  for 
which  It  Is  Intended:  It  emphasizes  the  improvement  which  can  be  obtained  by 
exclusion  of  a  particular  sample.  However,  beware  of  repeated  deletions.  The 
probability  levels  of  the  test  change  step-wise  with  each  application. 
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IN-MINEFIELD  EFFECTIVENESS  MEASURE  FOR  BREACHING  VEHICLES 


\ 


Mark  S.  Adams 

Systems  Employment  Branch,  Counteimine  Laboratory 
US  Arny  Mobility  Equipment  Research  and  Development  Oommand 
Fort  Bel  voir,  Virginia 

i  nnaoptmcw 

^  The  development  of'  realistic  models  Is  required  to  assess  the  military  worth 
of  countermine  systems  In  mine  warfare  scenarios.  Explicit  closed  form  solutions 
delineating  countermine  equipment  effectiveness  are  being  developed  to  become 
modular  components  of  a  more  complex  war  game  modelling' mine  warfare. 

This  report  develops  a  closed  solution  to  measure  the  effectiveness  of 
armored  vehicles  proceeding  through  cleared  lanes.  An  equation  is  derived  to 
determine  the  expected  nurber  of  mines  a  vehicle  will  encounter  In  a  scenario. 

The  expected  number  of  mine  encounters  is  used  to  calculate  a  measure  to 
compare  the  value  of  changes  In  tactical  methods  and  countennlne  materiel. 

A  discussion  of  tbs  applicability  of  the  effectiveness  measure  to  support 
mins  and  countermine  studies  Is  also  presented.  A  set  of  mine  warfare  situations 
are  formulated  as  an  example  of  the  ease  of  using  the  expression  derived  In 

this  report.  ^ - - - - - „ - - 

II  BACKGROUND 

In  June  1980,  US  AES  requested  MERADCCM  to  perform  analyses  determining 
what  marking  systems  could  be  used  with  mine-clearing  rollers.  MERADCCM  tasked 
HEL  who,  In  turn,  subcontracted  Armament'  Systems,  Inc.  and  the  final  report, 

"An  Investigation  of  Requirements  for  Cleared-Lane  Marking  Systems  (CLAMS)  for 
Hasty  Breaching  of  Minefields  with  Mine-Clearing  Rollers,"  was  completed  In 
March  198l.  Section  ,5.0  of  this  report,  "Assessment  of  the  Problems  Associated 
with  Traversing  and  Marking  a  Minefield,"  thoroughly  discusses  doctrine > 
literature  and  field  test  data,  and  reports  on  requirements  of  the  width  of 
a  cleared  lane.  The  requirements  establisned  in  sources  such  as  FM  90-7 
"Obstacles"  (which  states  a  4  meter  wide  vehicle  assault  lane  may  be  used  for 


the  hasty  breach),  and,  the  final  report  by  the  USA  Amor  and  Engineering  Board 
of  OT  II  testing  of  the  Mine-Clearing  Roller  (that  notes  the  width  of  the  lane 
cleared  by  one  tank  with  a  roller  is  Inadequate  to  allow  safe  tracking  by 
other  tanks  and  personnel  carriers),  are  not  driven  by  ,  or  directly  related 
to,  mobility  mission  requirements  or  effectiveness.  Sane  procedure  Is  necessary 
to  aid  translating  mission  requirements  Into  equipment  performance  requirements 
and  the  converse.  With  the  expression  developed  here,  postulated  systems. 
performance  functions  can  generate  values  of  in-minefield  effectiveness  as 
the  measure  of  the  expected  number  of  mines  a  vehicle  will  encounter. 

Ill  OBJECTIVE  AND  SCOPE 

The  objective  of  this  report  Is  to  derive  an  equation  to  calculate  expected 
mine  enoounters  of  vehicles  crossing  a  minefield,  and  describe  metnods  In 
computing  and  applications  of  results  of  the  equation  to  determine  ln-mlnefleld 
effectiveness  (IME)  measures  of  various  systems. 

The  mathematical  scope  of  the  derivation  extends  to  an  integral  calculus 
statement : 

/(density  function)  d(ARElA)  ■  units  (1) 

AREA 

For  the  application  here,  this  translates  to:  the  Integral  of  the  density 
function  In  mines  per  cquare  meter  over  the  are^  swept  out  by  a  vehicle  passing 
through  a  minefield  is  equal  to  the  number  of  mines  the  vehicle  will  encounter. 

This-  would  be  exactly  true  if  mines  were  a  continuous  phenomenon.  But  since 
they  are  point  located,  or  at  best  disjoint,  this  equation  is  an  approximation  ' 
to  the  expected  number  of  mine  encounters.  This  method  does,  not  calculate 
where  a  vehicle  will  encounter  a  mine,  only  the  expected  number  of  encounters 
that  are  found  in  the  area  used  in  the  calculation.  This  expected  value  is 
the  identical  concept  to  the  average,  or  mean  value. 


Actual  mine  location  will  vary  under  a  host  of  conditions  (mine  laying 
procedure,  minefield  layout,  etc.),  so  there  is  a  possiblity  that,  though  the 
expected  value  of  encounters  in  an  intended  path  is  positive,  actually  transversing 
the  minefield  will  result  in  no  mine,  encounters. 

Under  rather  standard  assumptions  of  randomness,' the  occurrence  of  mine 
encounters  can  be  treated  as  a  Poisson  process.  Arguements  to  independent  and 
identical  distributions  are  not  very  serious  because,  firstly,  the  calculation 
derived  is  an  approximation,  and  secondly,  the  performance  measure  relates 
to  the  probability  of  no  occurences,  so  the  memoryless  criteria  of  such  a 
process  is  robust.  According  to  a  Poisson  distribution,  the  probability  of  n 
occurences,  Pr(n),  given  that  the  expected  number  of  occurances  is  X,  is  given 
by  equation  2. 


Pr(n) 


(2) 


The  probability  of  encountering  no  mines  (n"0)  given  the  expected  number  of 
encounters,  X,  to  be  E(N)  is 


Pr(0  mines)  ■  e“E(N) 


(3) 


This  probability  is  the  in-minefield  effectiveness  measure. 


Included  in  the  IME  squat io^  derived  within  is  *  parameterized 
function  th&t  describes  the  vehicle  path  and  a  probability  distribution  function 
of  random  play  about  the  path.  The  resulting  product  of  functions  provides  a 
flexibility  In  the  quantifiable  description  of  countermining  situations.  Moreover, 
the  results  are  Immediately  and  Inexpenaivly  obtained  compared  to  simulations 
and  war  games  of  similar  scope. 

17  EERIVATION 


Let  i(x,y)  be  the  minefield  density  function  over  some  Cartesian  coordinate 

2 

system;  typical  units  for  such  a  funotion  are  mines  per  meter  squared  (M/m  ) . 

Take,  for  example,  a  situation  at  PIO  1.  In  this  example,  a  mechanically 

enplaoed  minefield  of  3  rows  of  pressure  AT  mines  with  an  inter-mine  spaoing  of 

four  meters  would  require  300  mines.  Since  the  area  of  this  minefield  is  20,000 

2 

square  meters,  tne  overall  density  of  the  mins f laid  is  0,015M/ta  .  An  alternate 
representation  of  this  minefield  could  partition  tne  mine  rows  into  separate 
minefields.  Using  tne  variables  defined  in  the  figure,  the  expected  number  of 
mines,  E(N),  s  vehicle  would  encounter  is  2  •  (ote-ite)  •  d  •  6.  The  difference, 
In  meters,  from  the  outer  track  edge  to  the  Inner  track  edge  (ote-ite)  is  one 
traok-wldth.  For  this  example,  E(N)*2 • (ote-ite}  -50 *0. 015-1. 5 * (ote-ite)  or 
1.5  times  a  track-width  of  the  vehicle  (in  meters)-.  This  calculation  is  simply 
the  area  swept  out  by  the  vehioles  tracks  tines  the  constant  minefield  density. 


Zn  integrel  form,  however. 


E®  ■  fo(|S&25  ®  - 


fw/2+ote 

w/2+ite 


i(x 


,y)  dx)  dy 


Since  4(x,y)  ■  0.015  M/m  fbr  0<x<w,  0<y<d,  we  have, 


ECO 


[o  ' 0.015(f 

fa  / / n  nie.O 


w/2-lte 

w/2-ote 


dx  + 


w/2+ote 
,  w/2+ite 


d xj  dy 


-  jQ  ((0.015*2)(ote-ite))dy 


■  0.030(ote-ite)d 

■  ( 1. 5 )( ote-ite) 


(4) 


(5) 
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Nomenclature  for  the  IME  Equation 


E  tz  _ 

E  .2  § 

ra  — -g 


•  o  T- 

E  H  M 
E  E 
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Figure  1.  STRAIGHT  LINE  PATH  THROUGH  A  MINEFIELD 


Uils  Is  the  identical  result  obtained  ear iler. 


Consider  now  that  the  vehicle  goes  through  the  minefield  on  something 
other  than  a  straight  line  path.  Define  P(m)  =  (X(m)jY(m))  as  the  parametrical 
representation  of  such  a  path,  where  m  ranging  from  0  to  1  relates  to  the 
amount  of  tne  mission  complete  (i.ev  m=0  is  the  start  of  the  mission,  m=l 


the  end).  The  areas  swept  out  by  tracks  of  a  vehicle  are  no  longer  simple 
rectangles,  as  shown  at  PIG  2.  Incorporating  this  parametrical  path  represen¬ 
tation,  equation  (3)  becomes 


E(N)  =  6(x,y)(dY/dm)dx  +. 


orx 

irx 


'5(x,y)(dY/dm)dxj  dm 


where  olx  **  outer  left-track  x  =  X(m)  -  ote/v/l+s2(m) 
ilx  =  inner  left-track  x  =  X(m)  -  lte/s/l+s^(m)' 
irx  »  inner  right-track  x  *  X(m)  +  ite/  s/l+s2(m) ' 
orx  -  outer  rlgfrt-track  x  *  X(m)  +  ote/ s/l+s^  (m)  ‘ 
y  »  Y(m)  +  s2(m)(X(m)-x) 


s(m) 


-dX/dm 

dY/am 


(6) 


Analytically,  this  integral  represents  the  summation  of  partitioned  rectangles 
that  make  up  the  area  traced  by  the  vehicle  tracks,  as  shown  in  FIG  3. 

Thus  far,  vehicles  are  restricted  to  following  the  prespecified  patns 
perfectly..  A  probability  distribution  function  of  play  about  the  path  is 
inserted  to  account  for  effects  of  vehicles  not  able  to  follow  precise  paths. 
This  play  function,  G(z,m),  is  dependent  upon  distance  from  the  path,  z, 
and  tne  mission  parameter  measure,  m.  The  final  form  for  the  expected  number 
of  mine  encounters  Is 


tOl)  =  jj  J*”ft(z,m) 


ilx 

olx 


<5(x,y )  (dY/dm)dx  + 


5(x,y)(dY/dm)dx  )  dz  dm 


Best  Available  Cop 
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Partitioning  of  the  Area 


Figure  3.  GRAPH  OF  PARAMETERIZED  PATH  THROUGH  A  MINEFIELD 


where  olx  "  X(m)  -  (ote-zVs/T+s^Tm? 


ilx  -  X(m)  -  (ite-z)/Vl+a2(m )' 
irx  ■  X(m)  +  (ite+z)/»/l+a2(m)i 
orx  -  X(m)  +  (ote+z)/>/l+s2(m)' 

y,  s(m)  as  defined  In  equation  (6).  (7) 

Once  E(N)  Is  calculated,  the  probability  of  the  vehiole  traversing  the  path 
P(m)  with  play  function  n(z,m)  through  a  minefield  density  <S(x,y)  without 
encountering  a  mine  is  given  by  equation  (3)  i 
Pr(0  mines)  ■  e*’®^ 

The  Pr(0  mines)  is  the  in-minefield  effectiveness  (IMS)  measure  for  breaching 
vehicles.  Being  a  probability,  the  IME  measure  will  always  fall  between  0  and  1, 
the  latter  designating  a  certainty  of  no  mine  encounters.  The  closer  the  IME  is 
to  one,  the  better  the  chances  a  vehiole  will  euooeasfully  cross  the  minefield. 

V  APPLICATION 

This  seotlon  presents  sample  applications  of  the  IMS  treasure  equation. 

The  Intent  of  this  seotlon  is  to  demonstrate  the  flexibility  of  the  rncdel;  it 
is  not  an  exhaustive  list  of  the  capability  of  the  procedure.  The  situations 
to  be  studied  will  dlotate  the  forms  of  three  functions :  minefield  density , 
vehicle  path,  and  path  play. 

Let  us  choose  as  a  problem  measuring  the  ability  of  follow-on  vehicles 
to  breach  a  minefield  first  oleared  by  a  single  lead  tank  equipped  with  a  mine- 
clearing  roller  And  oleared  lane  marking  system. 

A  minefield  density  function  to  represent  this  situation  must  be  formulated. 
Consider  the  mechanically  unplaced  minefield  of  300  mines  in  a  400  meter  by  50 

meter  rectangle  dlsoussed  earlier.  Before  any  neutralization,  the  density  could 

2 

be  taken  as  a  constant  0.0151'1/m  .  As  the  lead  tank  equipped  with  a  roller 
passes  through  the  minefield  and  detonates  mines ,  the  density  of  the  minefield 
is  lowered.  Assuming  perfect  capabilities,  and  a  straight  line  breach 
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perpendicularly  through  the  center  of  the  minefield,  the  density  of  the  minefield 
drops  to  zero  within  the  two  rectangles  traced  out  by  the  signature  of  the  roller 


banks.  A  representative  minefield  density  function  before  and  after  nuetralization 
is  shown  at  FXQ  4.  ‘The  aftez'-neutralization  minefield  density  function  is  used 
to  determine  the  expected  number  of  mines  encountered  by  follow-on  vehicles . 

The  algebraic  expression  of  this  density  function  is 


«C*.ir)  -[g; 


0  0.9l|x-200|<2.0 

015  otherwise,  0<x<400,  0<y<£0 


(8) 


Physically,  equation  (8)  models  a  lead  tank  that  crossed  the  minefield  at 


raid- front  (x-200  of  the  400  m  minefield),  with  2  mine-clearing  roller  banics  1.1m 
wide  separated  by  1.8  meters. 

lhe  second  of  the  three  IME  functions,  the  vehicle  path,  models  the  attempted 
path  of  follow-on  vehioles  making  full  advantage  of  neutralized  zones.  In  this 
Instance ,  the  intended  path  for  such  vehioles  is  to  retrace  the  straight  line 
path  of  the  lead  tank  guided  by  soma  marking  system.  The  parameterized  form  of 
this  path  la 

P(m)  -  (X(m),Y(m))  i  X(m)  -  200 

Y(m)  *  50  •  m  (9) 

At  the  start  of  the  mission,  the  vehicle  is  at  P(0)  which  is  (200,  0)  on 
the  Cartesian  system  employed.  By  the  end  of  the  mission  (m-1) ,  the  vehicle 
has  travelled  In  a  straight  line  to  (200,  50);  this  is  the  same • path  as  the 
roller  equipped  lead  tank. 

However,  dtie  to  many  conditions,  follow-cn  vehicles  cannot  exactly  duplicate 
the  lead  tank  path.  A  family  of  play  functions  is  postulated  and  implemented 
to  model  the  -ability  of  these  follow-on  vehioles  to  stay  on  the  intended  path. 

The  play  function  can  be  interpreted  as  the  capability  of  the  driver  of  a  follow-on 
vehicle  to  stay  oh  the  Intended  path  based  on  his  skills,  training,  driving  aid 
devices ,  and/or  marking  systems.  A  perfect  path  follow-on  vehicle  could  be 
thought  of  as  one  whose  play  never  strays  off  the  intended  trace  (i.e.,  the  center 
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o.’  the  follow-on  vehicle  exactly  retraces  the  center  of  the  path  of  the  lead 
vehicle).  As  conditions  drop  from  less  than  perfect,  follow-dn  vehicles  have 
higher  probabilities  to  stray  away  from  the  intended  path.  She  sample  play 
functions  used  in  this  application  are  at  FIG  5. 

The  discontinuous  nature  of  S(x,y)  and  several  of  the  n(z,m)  functions  lead, 
to  cumbersome  arithmetic  calculations.  To  ease  this  problem,  a  computer  program1 
was  written  to  readily  analyze  the  expressions.  Table  I  is  a  sumnary  of  sample 
applications  of  the  IME  methodology.  Column  four  of  this  table  showb  the  IME 
measures  for  the  postulated  systems.  The  effective  path  width  listed  in  Column 
five  is  the  sum  of  the  range  of  play  allowed  and  the  width  of  the  follow-on 
vehicle. 

The  IME  measures  change  significantly  over  the  set  of  play  functions.  For 
test  run  1,  the  IME  la  1.00,  msanlng  100*  chance  of  crossing  the  minefield  without 
mine  encounter.  This  is  a  reasonable  result,  for  this  trial  is  with  no  play; 
the  follow-on  vehicle  path  exactly  matches  the  lead  tank  path,  and  the  tracks 
6f  the  fbllow-on  vehicles  will  always  fall  between  the  bounds  of  the  safe 
zones  cleared  by  the  rollers.  Allowing  the  vehicle  to  sway  +  0.5  m  from  the 
perfect  lane  (trial  2)  drops  the  chance  of  encountering  no  minus  5  percentage 
points.  Normally  distributed  play  functions  perform  better  than  triangularly 
distributed  ones  of  equal  range  (trial  3  vs  5  and  4  vs  6)  because  the  normal 
distributions  have  a  greater  central  tendency  (they  hug  the  line  better)  than 
the  triangular  distributions. 

The  IME  measure  indicates  that  a  reduction  in  play  of  1  mater  betters  the 
probability  of  no  mine  encounters  by  0.13  (trial  6  compared  with  trial  5).  The 
chance  of  encountering  no  mines  junps  from  72%  to  85%  when  the  follow-on  vehicle 


1A  listing  of  this  program,  written  in  SUBSCRIPT  II. 5,  is  in  the  Appendix. 
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Test  Description 


Range  (m) 


Q  (z,m) 


Comment 


1  No  Play.  0.0 


2  Triangular  Distribution  1.0 


3  Ti Angular  Distribution  2.0 


4  Triangular  Distribution  3.0 


5  Normal  Distribution  2.0 


6  Normal  Distribution  3.0 


7  Normal  Distribution  2.0 


-1.0  0  1.0  Z 


meters 


-1.0  0  1.0  Z 


meters 


-1.0  0  1.0  Z 


-1.0  0  1.0  Z 


meters 


Perfect  Tracing 
of  Path. 


^  =  0.0 
o  =  1/3 


M  =  0.0 
o  =  1/2 


M  =  0.5 
o  =  1/3 
(Right  Bias) 


Figure  5.  SAMPLE  PLAY  FUNCTIONS  USED  IN  THE  IME  APPLICATIONS 
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Table  i.  SUMMARY  OF  SAMPLE  APPLICATION  OF  THE  IN-MINEFIELD  EFFECTIVENESS  MEASURE 


restricts  its  deviation  about  the  path  a  half  meter  on  each  side.  This  translates 
to  a  potential  benefit  due  to  increased  survivability.  This  benefit  can  be 
realized  through  increased  driver  skill,  improved  training,  and/or  cleared-lane 
marking  system,  any  approaches  to  result  in  less  play  of  follow-on  about  a  clear 
path. 

The  following  section  describes  a  study  done  to  evaluate  marking  systems 
and  how  IME  could  have  been  employed  to  achieve  meaningful  results. 

VI  DISCUSSION 

The  Concept  Evaluation  Program  of  CLAMS  (3  December  1981,  TRADOC  ACN  52725) 
compared  the  operational  performance  of  chemiluminescent  candles  to  highway 
safety  flares  in  marking  a  breach  through  a  minefield.  Trial  runs  were  scored 
as  successful  if  a  vehicle  stayed  within  predetermined  path  widths  88%  of  the 
time  during  a  breach.  Measurements  were  taken  as  the  vehicle  passed  each 
marker.  Results  from  this  test  were  non-conclusive.  Only  7  of  203  attempts 
by  M60  tanks  to  negotiate  a  4  meter  path  were  successful.  The  binary  nature 
of  the  outcome  of  a  trial  (labeled  succeos  or  failure)  contributed  greatly 
to  the  insensitivity  of  the  results  of  the  field  experiments.  Moreover,  the 
outcome  labels  had  little  tu  do  with  mission  sucoess  or  failures  of  vehicles 
breaching  hypothetical  minefields  as  those  simulated  by  the  tests.  Failure  to 
maintain  a  four  meter  path  in  the  test  did  not  directly  equute  to  failure  to 
breach  the  simulated  minefield,  and  the  same  is  true  for  success.  There  was 
no  reference  to  a  real  military  worth. 

The  IME  equation  provides  the  means  to  combine  minefield  density,  vehicle 
track  signature  and  path  into  a  quantitative  assessment  of  military  worth.  The 
exomple  results  show  the  gains  in  terms  of  higher  survivability  by  achieving 
narrower  vehicle  path  tracings.  Field  experiments  takan  to  measure  vehicle  path 
functions  can  be  translated  into  quantitative  measuros  attributable  to  the 
military  worth  of  the  marking  systems  through  the  IHE. 
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V  CONCLUSIONS 


The  IME  measure  is  an  easily  calculated,  yet  sensitive  indicator  of  the 
performance  of  mine  and  countermine  systems.  Elements  of  mine/countermine  systems 
can  be  modelled  by  the  various  data  and  function  inputs  to  the  IME  equation,  as 
listed  in  Table  II.  Complex  functions  can  be  evaluated  with  a  computer  program 
specifically  designed  to  solve  IME  equations.  The  IME  measure  is  a  useful 
index  because  it  translates  system  performance  characteristics  of  alternative 
mine /countermine  systems  into  survivability  figures.  The  IME  process  can 
quantify  benefits  of  new  developments,  whether  organizational,  operational,  or 
materiel  in  nature.  IME  measures  can  also  be  used  as  input  to  higher  level, 
larger  scope  war  games  where  previous  data  were  randomly  generated  or  estimated. 

Examples  of  other  uses  of  IME  are: 

a.  Mixed  mine  type  minefield  effectivneas.  The  different  mine  type 
densities  and  corresponding  track  or  vehicle  signatures  can  initially  be 
aeparated  out  and  later  combined  for  an  aggregate  effectiveness  measure. 

b.  Smart  mine  design  parameters.  The  parameterized  path  function  can  be 
time  normalized  and  probability  of  mine/ vehicle  encounter  based  on  duration  of 
axposure  as  well  as  area. 

c.  Countermine  systems  mix  analyses.  Single  systems  and  combination 
can  be  studied. 

d.  Hide  area  countermine  systems  analyses.  Hypothetical  countermine 
system  performances  can  be  compared  as  to  how  well  they  neutralize  threat 
minefields  for  follow  on  vehicles. 
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MXKE/COIMCTMPE  ELEMENT 
MINE  EETECTION 

MINE  NEUTRALIZATION 

MARKING  SYSTEMS 

TERRAS 

FITZINO/TANK  SIONATUPE  D7IERACIT0NS 

MINE  LAYING  PATTERN 
TRAINING  AND  DOCTRINE 


TABLE  II.  3ME  EQUATION  OOMPONOTTS  MODELLING  M1NE/COUNIERMINE  ELEMENTS 
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ABBREVIATIONS 


AT 

CLAMS 

FAE 

HEL 

IME 

OT 

USAES 

Wr£ 


Antitank 

Cleared- Lane  Marking  System 

Fuel-Air  Explosive 

Human  Engineering  Laboratory 

In-Minefield  Effectiveness 

Operational  Testing 

U.S.  Army  Engineer  School 

meters 

mines  per  meter  squared 


d 

w 

m 

x 

y 

z 

lte 

ote 

llx 

irx 

olx 

orx 

P(m) 

X(m) 

Y(m) 

dX/dm 

d*/dn 

s(m) 

E(N) 

Pr(n) 

«(x.y> 

x 

u 

0 

B(z,m) 


SYMBOLS 

depth  of  minefield  (meters) 
width  of  minefield  (meters) 
mission  parameter  o<jh<1 

position  within  minefield  along  the  width  (meters) 
position  within  minefield  along  the  denth  (meters) 
deviation  from  prescribed  path  (meters) 

Inner  track  edge;  the  distance  from  the  center  of  a  vehicle 
to  the  inner  track  edge  (meters) 

outer  track  edge;  the  distance  from  the  center  of  a  vehicle 
to  the  outer  track  edge  (meters) 

Inner  left  traok  edge  integrand 

inner  right  traok  edge  integrand 

outer  left  track  edge  integrand 

outer  right  traok  edge  integrand 

parameterized  path  function 

X  oonponent  of  path 

Y  oonponent  of  path 

first  derivative  or  X(m) 

first  derivative  of  Y(m) 

slope  of  vehicle  orientation  at  m 

the  expected  value  of  the  number  of  mine  encounters 

the  probability  of  n  occurences 

delta,  the  minefield  density  function 

lambda,  the  mean 

mu,  the  mean  of  a  normal  distribution 

signs,  the  standard  deviation  of  a  normal  distribution 

omega,  the  play  function 

infinity 
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APPEND!? 


COMPUTER  PROGRAM  LISTING  FOR  THE  IME  EQUATION 


INDEX 


,1 

y\ 


v| 

m 


'*i 


LISTING 

TEST  RUN  1  PLAY  FUNCTION  AND  INPUT  FILE 
TEST  RUN  1  OUTPUT 

TEST  RUN  2  PLAY  FUNCTION  AND  INPUT  FIXE 
TEST  RUN  2  OUTPUT 

TEST  RUN  3  PLAY  FUNCTION  AND  INPUT  FILE 
TEST  RUN  3  OUTPUT 

TEST  RUN  1»  PLAY  FUNCTION  AND  INPUT  FILE 
TEST  RUN  H  OUTPUT 

TEST  RUN  5  PLAY  FUNCTION  AND  INPUT  FILE 
TEST  RUN  5  OUTPUT 

TEST  RUN  6  PLAY  FUNCTION  AND  INPUT  FILE 
TEST  RUN  6  OUTPUT 

TEST  RUN  7  PLAY  FUNCTION  AND  INPUT  FIIZ 
TEST  RUN  7  OUTPUT 


A2  -  A4 
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ROBUST  RANGE  MEASUREMENT  PREPROCESSING 


William  8.  Agaa  and  Robert  H.  Turner 
Mathematical  Sarvlcea  Branch 
Data  Sciences  Division 
National  Range  Operatlona  Directorate 
US  Army  White  Sands  Missile  Range 
White  Sands  Missile  Range,  New  Mexico 


ABSTRACT 

The  RM8/MTTS  instrumentation  system  located  at  MacGregor  Range  is  a  range 
measuring,  multiple  target  tracking  system.  In  order  to  obtain  a  vehicle 
trajectory  from  this  system,  the  range  measurement  from  several  recalvets 
are  processed  by  least  squares.  Since  the  measured  vehicle  trajectories 
are  often  low  altitude,  the  resulting  nonlinear  least  squares  aquations 
are  ill-conditioned.  In  addition,  this  measurement  system  is  plagued  by 
outliers,  sometimes  by  dense  burst  of  outliers.  The  combination  of 
111-condltlonlng  and  outliers  is  lethal  and  attempts  to  robustlfy  the 
nonlinear  least  squares  processing  have  failed.  An  alternative  is  to 
preprocese  each  of  the  range  measurement  sequences,  eliminating  the 
outliers  and  replacing  them  if  nocaasary.  Each  aequance  of  range  measure¬ 
ments  is  preprocessed  by  robustly  fitting  a  cubic  spline  using  iteratively 
rewaighted  laeat  squares.  Due  to  the  nature  of  spline  fitting  and  the 
possible  dense  burets  of  outliers,  the  choice  of  a  good  set  of  initial 
weights  for  use  in  the  iteratively  revslghted  least  squares  is 
important  to  the  efficiency  of  the  method.  These  initial  weights  are 
determined  using  robust,  local  fitting  techniquae.  Several  robust  techniques 
have  been  tested  for  this  local  fitting  application.  The  robust  spline 
preprocessing  is  illustrated  with  some  especially  troublesome  data  sequences 
and  the  relative  performance  of  several  robust  methods  for  choosing  the 
initial  weights  is  compared. 


INTRODUCTION 


The  RMS/MTTS  Instrumentation  system  located  at  MacGregor  Range  Is  a  range 
measuring*  multiple  target  tracking  system.  In  order  to  obtain  a  vehicle 

trajectory  from  this  system,  the  range  measurements  from  several  receivers 

* 

are  processed  by  least  squares.  Because  the  measured  vehicle  trajectories 
of  Interest  are  often  low  altitude  and  because  of  the  geometry  of  the  re¬ 
ceiving  stations,  the  resulting  nonlinear  least  squares  equations  are  often 
Ill-conditioned.  In  addition,  this  measurement  system  Is  subject  to  out¬ 
liers,  sometimes  dense  bursts  of  outliers.  This  combination  of  111 -condition¬ 
ing  and  outliers  Is  lethal  and  our  attempts  to  robustlfy  the  nonlinear  least 
squares  estimation  process  have  failed.  An  alternative  Is  to  preprocess  each 
of  the  range  measurement  sequences.  Identifying  the  outliers,  and  replacing 
them  If  necessary.  The  Ill-conditioned  least  squares  preblehk^an  then  be 
troted  without  being  troubled  by  outliers.  x. 

X. 

Suppose  we  preprocess  the  measurement  sequence,  R(t1),  1  ■  1,N.  For  typical 
aircraft  trajectories  the  measurement  rate  Is  10/sec  with  a  time  of  Interest 
of  40  -  120  sec  so  that  N  Is  often  In' the  range  400  -  1200.  The  purpose  of 
the  preprocessing  may  be  to  detect  outliers,  to  precompute  measurement  variances 
for  future  least  squares  processing,  or  to  synchronize  several  different  dis¬ 
crete  measurement  sequences.  The  preprocessing  of  the  range  measurement 
sequence,  R(t^),  Is  done  by  fitting  a  cubic  spline  to  the  discrete  measurements 
using  Iteratively  reweighted  least  squares  (IRWLS).  Specifically,  at  the  k— 
Iteration  we  minimize, 


■  \  »rvv> 


(k), 


(D 
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(L) 

where  ( * )  are  the  cubic  B-splInes  and  b^  '  are  the  spline  coefficients  to 

Ik) 

be  estimated.  The  weights,  Wj  '  are  computed  from  the  Hampel  ^-function 
using  the  spline  fit  from  the  (k-1)—  Iteration. 

fottj)  -  lb\km})  B^tj) 


u(k)  . 

WJ 


.k-1 


R(t 


fa 


TT 
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where 
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estimates  the  dispersion  In  the  residual,  R(t.)  -  7  bjK“ 1 (t .) .  The 

/i.  n  \  1 


(k-1), 


value  of  S 


(k-1) 
j 


can  be  computed  either  locally  or  globally  from  the  residuals 


at  the  (k-1)—  Iteration.  The  dispersion  Is  a  MAD  estimate  obtained 


from 


s!k-1)  -  median  |R(tm)  -  7  bjk_1 )B1 (tm) |/.6745  (4) 

J  meTj  ' 

If  the  set  Tj  Is  In  some  sense  the  set  of  points  close  to  t^,  the  estimate 
Is  local.  If  the  set  Tj  Is  the  set,  Tj  »  {tjm  ■  1,N}  the  estimate 
is  global.  For  the  present  application  only  the  global  estimate  S^"1^ 

will  be  used.  If  a  very  long  data  sequence,  say  about  one  hour,  a  local 
estimate  would  probably  be  preferable  to  the  global  estimate. 

CHOICE  OF  KNOTS 

Let  {T^ ,  1  ■  1,M)  be  a  set  of  knot  times.  These  knot  times  are  used  to  define 
the  cubic  B-splInes,  (tj ) .  Of  most  Importance  In  the  choice  of  the  knot  times 
Is  their  spacing,  which  determines  the  ability  of  the  cubic  spline  to  fit  the 
data.  However,  for  each  additional  knot  time  there  Is  one  additional  spline  co¬ 
efficient  to  be  estimated,  thus  increasing  the  computational  load.  Thus,  we  want 
to  have  as  few  knots  as  possible  and  the  rules  for  their  choice  simple  and  yet  be 
able  to  adequately  represent  the  data.  With  this  simple  philosophy  for  selecting 
knots  we  will  try  to  assign  a  fixed  number  of  data  points,  NPTO,  to  each  knot  In¬ 
terval.  The  first  four  knots  are  placed  at  the  first  data.  The  last  knot  Inter¬ 
val  may  have  more  than  NPTO  points  but  fewer  than  2  •  NPTO  data  points.  If  there 
Is  a  large  time  break  in  the  data,  a  knot  Is  placed  at  the  beginning  and  end 


-■ 


*  nj  ? 


of  the  time  break.  The  Interval  between  these  two  knots  has  zero  data  points 
and  the  Interval  Immediately  precedinq  the  time  break  may  have  more  than 
NPTO  points  out  fewer  than  2  •  NPTO  points.  If  immediately  after  a  time  break 
there  4s  a  second  time  break  before  NPTO  points  have  been  read,  the  few  (less 
than  NPTO)  points  read  between  the  two  time  breaks  are  discarded.  If  a  time 
break  occurs  while  reading  points  for  the  first  knot  Interval,  the  few  (less 
than  NPTO)  points  are  discarded  and  the  first  four  knots  repositioned  at  the 
first  data  time  after  the  time  break.  If  a  time  break  occurs  during  the  last 
Interval,  the  portion  of  the  last  Interval  contiguous  to  the  previous  interval 
Is  kept  and  the  remainder  of  the  points  In  the  last  Interval  are  discarded. 

If  there  are  at  least  NPTO  points  kept,  these  points  form  the  last  Interval. 

If  there  are  less  than  NPTO  points  kept,  these  points  are  appended  to  the  pre¬ 
vious  Interval  so  that  the  number  of  knot  Intervals  Is  reduced  b„'  The 

time  difference  between  successive  data  points  which  Is  used  to  define  a  time 
break  is  named  FITBRK.  FITBRK  Is  dependent  on  the  sample  rate.  The  time  dif¬ 
ference  between  successive  data  points  used  to  define  a  time  break  In  the  first 
and  last  knot  Intervals  Is  FITBRK/5.  This  smaller  value  Is  used  In  the  first 
and  last  Interval  because  It  Is  critical  to  obtain  a  good  fit  In  these  Intervals. 
The  flow  chart  on  the  following  pages  more  clearly  defines  the  logic  for  select¬ 
ing  the  knot  times.  The  following  define  the  variables  in  the  flow  chart: 

NOTS  «  number  of  interior  knots  R(*)  ■  array  of  range  measurements 

KR  *  number  of  knot  intervals  NPTS(*)  ■>  array  of  point  counts  for 

knot  Intervals 

TT( * )  ■  array  of  knot  times 

IBRK  -  logical  denoting  the  occurrence 
NC0UN  »  total  data  point  count  of  a  time  break 

T( • )  ■  array  of  data  times  STA  ■  data  start  time 
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00  FOR  i  ■  1,  N 


t  'J&OrtXRf  xS^ftSftu  w.-* 


NOTS  i  r 


THEN  ETA  •  TT(N0TS  +  3)H 
NCOUN  -  NCOUN  -  NPI 


TT(N0TS  +  3)  ■  TL  ♦  EPS  j 

NPTS  (NOTS  -  1)  •  NPTS(NOTS  -  1  )♦  HP; 
ETA _ _ ■  TL _  1 

KR  -  NOTS  -  i 

INOX  -  NCOUN  -  NPTS(KR)  - - 

TL  •  T(INOX  +  1) 


DO  FOR  K  ■  2,  NPTS ( KR ) 


INDX  +  K)  2.  . 2*FITBRB 


K-1  £  NPTO>— TH-E-lLi  NOTS  ■  N0T5-1  k<$OYS  THEh< 


NPTS(KR)  •  K-1 


TT(NOTS  +  3)  ■  TL  +  EP$I 
\m  ■  TL _ ) 


KR  •  KR  -  1 

NPT$(KR)  -  NOTS(KR)+K-l 


TL  •  T(INDX)  ,+  K 


THE  LEAST  SQUARES  NORMAL  EQUATION'S 

At  the  k—  Iteration  of  the  fitting  procedure  the  weighted  sum  of  squares 


N 

A 


,00 


-p; 


to 


W>’ 


(5) 


It  minimized.  The  least  squares  normal  equations  are  obtained  by  differentiating 
(5)  with  respect  to  bj*^.  The  least  squares  normal  equations  are 


Z  w!k)  B ( 1 4 )  BT(cJ  b(k)- 
J-l  3  3  3 


I  »[k)  B(tJ  R(tO 
j-1  3  3  1 


(6) 


where  BT(tj)  Is  the  vector  of  cubic  B-spHnes 

BT(tj)  -  CBxttj)  Ba(tj )  -  •  -  Bn(tj>3  (7) 

Due  to  the  nature  of  the  B-spllnes  the  positive  definite  matrix  on  the  left  of 

(  6)  Is  banded  with  three  bands  above  and  below  the  main  diagonal.  To  conserve 

storage  the  four  distinct  diagonals  of  this  matrix  are  stored  as  columns  of  a 

''(k! 

vertical  matrix.  The  dimension  of  the  vector  b'  '  Is  MOTS  ♦  2  whero  NOTS  Is 
the  number  of  Interior  knots,  Tho  banded  least  squared  normal  equations  are 
solved  by  a  banded  Cholesky  decomposition  algorithm.  The  turns  of  both  sides  of 
(6  )  are  performed  sequentially  so  that  all  of  the  ranges  and  weights  are  not 
needed  In  core  simultaneously.  The  IRWLS  can  be  continued  for  a  fixed  number 
of  Iterations  or  until  the  fit  has.  converged. 


INITIAL  WEIGHTS 


In  many  situations  the  IRWLS  procedure  works  successfully  when  all  of  the 
Initial  weights  are  set  to  one,  l.e.,  the  Iteration  Is  started  with  an  ordinary 
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unweighted  least  squares  solution.  He  have  found  that  the  use  of  the  unweleht- 
ed  least  squares  start  will  usually  result  In  convergence  of  the  IRWLS  cubic 
spline  to  a  good  fit  with  outliers  correctly  Identified,  but  that  many  fewer 
Iterations  are  required  If  a  robust  choice  of  Initial  weights  Is  used.  When 
outliers  are  present  In  either  the  first  or  last  Intervals,  the  choice  of 
Initial  weights  In  these  Intervals  Is  most  Important. 

The  Initial  weights  for  the  robust  cubic  spline  fit  are  chosen  on  a  localized 

th 

basis.  Let  R(t^),  1  ■  1,  NP1S(K)  be  the  range  measurements  In  the  K—  knot 
Interval.  To  determine  the  weights  wj°),  1*1,  NPTS(K)  In  the  K—  Interval  a 
linear  curve  Is  robustly  fitted  to  the  measurements  In  the  Interval.  Several 
methods  for  robustly  fitting  the  linear  curve  have  been  tried.  Including  the 
nested  median  method  of  Siegel  [  1  ],  the  method  of  Thell  [  2],  a  modified  Thell 
method,  and  an  M-estlmate  using  a  Hampel  <J»-funct1on.  Most  methods  performed 
about  equally  well  on  the  data  sequences  tested..  The  results  of  some  of  thesu 
tests  are  given  In  Appendix  A.  Because  of  Its  simplicity,  the  modified  method 
of  Thell  was  selected  for  routine  application.  This  method  is  described  in  the 
following  paragraph. 

Let  R  be  the  median  of  the  observations  In  the  K—  knot  interval  . 


R  ■  median  {R(t,)> 

1  -  1 ,  NPTS ' (K) 


(8) 


Let  t  be  the  time  corresponding  to  R.  The  median  R  can  be  represented  as  the 
average  of  two  observations, 


R  •  <R<W  +  R<‘m!))/2  (?> 

where  mi  ■  m2  If  NPTS(K)  Is  odd.  Define  the  set  of  slopes  {S^} 
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1-1,  NPTS(K) 
1  e  mt,  m2 


(10) 


,  .  R(t^)  -  R 


Let  i  be  the  median  of  the  slopes, 
i  ■  median  {S^} 

(  •  1.  HPTS(K)  („) 

1  i  nil, 


Let  r^  be  the  residual, 

^  *  R(t,)  -  -  t).  1  •  1,  NPTS(K) 

Let  r  be  the  median  of  these  residuals, 

r  -  median  {r^} 

1-1,  NPTS(K'- 


02) 


(13) 


Now  compute  the  residuals. 


r1  *  ri  •  r*  1 


1,  NPTS(K) 

j(  o ) 


(14) 


The  Initial  weights,  Wj  ,  are  computed  from  these  residuals  using  a  Hampel 
function. 


,(o) 


ft) 


NPTS(K) 


(15) 


Where  s^  Is  tho  robust  dispersion  parameter, 

sk  ■  median  { | r1 1 >/ . 6745  (16) 

Since  the  main  concern  In  setting  the  Initial  weights  Is  to  protect  the  cubic 
spline  fit  from  the  gross  outliers,  the  break  points  of  the  Hampel  ^-function 
are  set  at  a  •  2,  b  ■  3,  c  ■  4. 
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SOME  EXAMPLES 


Several  hundred  data  pastes  have  been  run  with  the  fitting  procedure  described. 
Slnee  there  Is  an  average  of  maybe  five  receivers  on  each  data  pass,  the 
robust  preprocessing  method  described  has  been  used  on  more  than  one  thousand 
measurement  sequences.  The  method  has  performed  successfully  on  all  of 
these  sequences.  Mott  of  these  sequences  are  rather  uneventful,  having  only 
a  few  Isolated  outliers.  There  have  been  some  sequences  which  have  some 
rather  dense  bursts  of  outliers.  These  sequences  best  Illustrate  the  ability 
of  the  method  described  to  detect  outliers.  Fig  1  presents  a  range  measurement 
sequence  and  Fig  2  the  robust  cubic  spline  fit  to  this  sequence.  Note  that  the 

outliers  In  Fig  1,  which  have  been  darkened,  occur  In  many  sizes.  The 
outliers  at  the  top  of  the  graph  were  added  by  hand  since  they  all  occurred 
far  off  scale  at  the  top.  The  sequence  of  Fig  1  has  about  10*  outliers. 

All  outliers  have  been  successfully  detected  and  removed  by  the  robust  spline 
fit.  The  measurements  In  Fig  1  have  two  dense  burst  of  outliers,  one  In  the 
Interval  (62356.6,  62359.5)  and  another  In  the  Interval  (62367.2,  62375.4). 

The  measurement  sequence  In  Fig  3  has  outlier  bursts  In  the  Intervals  (62358.4, 
62362.6),  (62374.4,  62376.4),  and  (62379.7,  62382.4).  The  sequence  In  Fig  3 
has  about  15*  outliers.  The  sequence  In  Fig  5  has  bursts  of  outliers 
during  the  Intervals  (63117.8,  63122.6)  and  (63128.2,  63131.5).  Any  points 
away  from  the  main  curve  should  be  considered  outliers  In  Figs  1,  3,  and  5. 

Note  also  In  Figs  1,  3,  and  5  that  there  are  time  breaks  In  the  measurement 
sequences,  another  Important  consideration  In  preprocessing.  The  cubic 
spline  fit  to  the  sequence  of  Fig  1  Is  given  In  Fig  he  cubic  spline 
fit  to  the  sequence  of  Fig  3  Is  given  In  Fig  4  and  the  cubic  spline  fit 
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to  the  measurement  sequence  In  Fig  5  Is  given  In  Fig  6.  The  knot  Intervals 
In  Figs  2,  4,  €  are  designed  to  contain  twenty  data  points.  Note  that  some 
of  the  time  breaks  have  been  filled  with  fitted  data  points.  The  filling 
of  the  time  breaks  Is  controlled  by  the  length  of  the  time  breaks  In 
relation  to  the  sample  rate  and  the  proportion  of  outliers  found  In  a 
knot  Interval.  The  robust  cubic  spline  preprocessor  has  deleted  all 
outliers  from  the  measurement  sequence,  generated  measurements  during  the 
time  breaks  as  desired,  and  synchronized  different  measurement  sequences 
If  desired.  In  addition  the  measurement  variances  are  available  for 
further  processing.  The  IRWLS  cubic  spline  fit  converged  In  3  -  4 
Iterations  for  the  examples  displayed.  This  fairly  rapid  convergence  Is 
dependent  on  a  robust  method  for  choosing  good  Initial  weights.  Surprisingly, 
the  IRWLS  cubic  spline  Iteration  for  these  examples  also  converges  using  an 
unweighted  least  squares  start,  but  at  the  expense  of  more  Iterations. 

For  the  measurement  sequences  displayed  here  the  IRWLS  cubic  spline  fit 
converged  In  7  -  8  Iterations  using  an  unweighted  least  squares  start. 

Thus,  at  least  In  these  examples,  a  good  choice  of  the  Initial  weights 
results  only  In  a  significant  Improvement  In  computing  efficiency  and  not  In 
an  Improvement  of  fit.  Besides  a  good  selection  of  Initial  weights,  another 
Important  choice  Is  the  number  of  data  points  per  knot  Interval,  NPTO. 

NPTO  must  be  large  enough  so  that  Is  likely  that  only  a  fraction,  say  less 
than  one  fourth  of  the  data  points  In  any  Interval  will  be  outliers.  On 
the  other  hand,  If  NPTO  Is  too  large,  the  robust  linear  curve  fit  may  not 
he  a  good  enough  representation  of  the  variation  of  the  data  In  the  Interval. 
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FIGURE  5 
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APPENDIX 


This  appendix  describes  several  methods  of  choosing  the  Initial  weights 
for  the  robust  cubic  spline  preprocessing  and  compares  the  results  of  using 
these  methods  on  several  data  sets.  Each  of  these  methods  robustly  fits  a 
linear  curve  In  each  of  the  knot  Intervals  and  then  computes  the  Initial 
weights  from  the  curve  fit  residuals  using  a  Hampel  ^-function.  Let 
R(t^ ) ,  tj  ■  1 ,  NPTS(k)  be  the  range  measurements  In  the  k™  knot  Interval. 

Thell  Method 


Define  the  slopes  s^j 


Rttj)  -  R^) 


J»1 


Let  i  be  the  median  of  these  slopes. 


5  ■  median  {s4<> 
1.J  1J 
J>1 


Define  the  residuals  r^, 

-  R^)  -  s  tt 

Let  r  be  the  median  of  the  residuals,  r4 

* 

?  ■  median  *r4) 

1.»l,NPTS(k)  1 

Then  the  residuals  r1  ■  r1  -  r  are  used  to  compute  the  Initial  weights  with  a 
Hampel  ^-function. 


Nested  Medians 


Nested  or  repeated  medians  Is  a  robust  regression  method  recently  described 
by  Siegel  [l].  Siegel  shows  that  tMs  method  has  the  highest  breakdown 
method  of  any  known  method.  This  method  Is  particularly  easy  to  apply  for 
a  linear  fit.  It  Is  similar  to  the  Thell  method  and  modified  Thell 
Method  already  described. 


Define  the  slopes  s^j, 
R(tJ  -  R(t^) 


* 4 /  *  1 


Define  s^  by 


»1 


median  {$..} 
J-l.NPTS(k) 

J-1 


(A-l) 


(A-2) 


and  further  let  5  be  defined  by 


i  «  median  (s. } 

l-l.NPTS(k)  1 


Similarly,  let  a^j  be  the  Intercepts 


“U  " 


R(tg)tj  -  R(tj)t1 


’  l1 


J-1 


Define  a^  as 


i.  "median  {a..} 

1  J-1,  NPTS(k)  1J 

4-1 


(A-3) 


(A-4) 


(A-5) 
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and  further  define  a  by 


a  •-  median  <a4  >  (A-6) 

l-l.NPTS(k)  1 

Let  r^  be  the  residuals 

r1  -  R(t1 )  -  a  -  i  t^  1-1 .  NPTS(k)  (A-7) 

The  weights  are  computed  from  these  residuals  using  a  Hampel 
♦-function. 


The  following  data  sets  were  taken  from  the  knot  intervals  of  the  data 
sequences  used  previously  to  Illustrate  the  application  of  the  robust 
range  measurement  preprocessing.  The  first  data  set,  shown  In  Fig  A1  is 
taken  from  the  measurement  sequence  given  In  Fig  1.  The  measurements  In  this 
set  are  from  the  time  Interval  62356.6  -  62359.5.  The  second  data  set, 
shown  In  Fig  A2  Is  taken  from  the  measurement  sequence  In  Fig  5.  The 
measurements  In  this  set  are  from  the  time  Interval  63128.2  -  63131.6. 

In  each  of  the  data  sets  the  weights  are  calculated  from  the  residuals  r^  by 


W<°> 

W1 


) 

mm 

) 


(A-8) 


where  ♦(•)  Is  a  Hampel  f-functlon  with  breakpoints  2.,  3.,  4.  In  both  of 
those  date  sets  there  are  eight  outliers  In  the  sample  of  twenty.  Each  of 
the  robust  linear  methods  seem  to  have  no  difficulty  In  Identifying  the 
outliers  In  these  data  sets. 
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ABSTRACT 


In  development  of  prototype  camouflage ,  performance  characteriatice  are 
determined  by  field  evaluation*.  The  ultimate  camouflage  being  that  of  a 
target  with  no  reatrictiona  on  time  or  manpower  in  its  erection  or  retraction, 
uaing  standard  materials  and  methods.  This  ultimate  condition  ia  not  normally 
measured  in  field  studies  where  prototype  camouflage  is  evaluated  against  a 
base  line  target.  Percentages  of  camouflage  improvement  in  detection  and  iden¬ 
tification  rates  by  the  prototype  over  base  line  conditions  are  identified. 

This  paper  concerns  a  reel  field  study . designed  to  allow  the  camoufleur  to 
measure  relative  improvementa  of  prototype  camouflage  against  both  base  line 
end  ultimate  conditions.  The  camoufleur  then  ascertains  true  values  of  the 
prototype  camouflage,  or  How  Good  is  Cood. 


1.0  INTRODUCTION 


The  development  of  camouflage  involvea  many  phases  and  evaluations.  In 
all  cases,  the  developers  goal  is  to  produce  the  best  camouflage  possible  within 
the  restraints  of  time  and  manpower.  One  method  of  camouflage  evaluation  is 
through  the  conduct  of  field  teats.  The  resulting  data  is  analyzed,  enabling 
the  developer  to  determine  the  effectiveness  of  his  product.  The  problem  with 
this  method  of  evaluation  is  that  while  it  provides  a  good  estimate  of  the 
prototype  camouflage  performance,  as  compared  against  the  base  line  camouflage 
condition,  no  information  is  available  for  a  comparison  against  the  ultimate 
camouflage  condition.  The  ultimate  camouflage  condition  is  defined  as  having 
no  restrictions  on  time  or  manpower  in  its  erection  or  retraction,  using  stan¬ 
dard  materials  and  methods.  Such  a  multiple  comparison  would  allow  the  develop¬ 
er  to  determine  the  virtues  of  additional  prototype  refinement  to  approach 
the  ultimate  camouflage  condition.  This  paper  concerns  a  real  field  study  de¬ 
signed  to  allow  the  developer  to  objectively  measure  the  relative  effectiveness 
of  prototype  camouflage  against  both  the  base  line  and  the  ultimate  camouflage 
conditions . 

2.0  TEST  SITE  AND  EQUIPMENT 

2.1  Test  Site 

•  The  test  site  was  located  at  the  USAF,  Avon  Park  bombing  Range  near 
Avon  Park,  Florida  (60  miles  south  of  Orlando,  Florida).  This  range  had  an 
average  elevation  of  68  feet  above  mean  sea  level,  and  is  flat  in  topography 
with  mixed  Oak,  Pine,  and  Palmetto  tree  hammocks  dispersed  in  sawgrasa.  The  land 
is  primarily  swampy  in  nature  and  displays  a  light  green  color  throughout. 

2.2  Test  Equipment 

The  equipment  tested  consisted  of  two  trailers  with  a  prime  mover 
and  were  over  40  feet  long,  7  feet  high  and  8  feet  wide.  The  equipment  was 
tactically  emplaced  along  a  tree  line  (one  was  emplaced  outside  the  tree  line 
and  one  was  partially  concealed  by  the  treeline). 

3.0  CAMOUFLAGE  CONDITIONS 

3.1  Pattern  Painted  (Base  Line) 

The  two  test  items  were  pattern  painted  the  tropic  color  blend  in 
accordance  with  TC5-200*'.  This  color  blend  consisted  of  45  percent  forest 
green,  45  percent  dark  green,  5  percent  light  green  and  5  percent  black. 

3.2  Pattern  Painted  with  the  Addition  of  Prototype  Camouflage 

The  second  camouflage  condition  was  achieved  by  the  addition  of 
camouflage  kits  to  the  two  patterned  vehicles.  These  camouflage  kits  were 
constructed  of  special  supports,  and  modified  lightweight  net  screening. 
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3.3  Pattern  Painted  with  the  Addition  of  Standard  Camouflage  Screens 
and  Techniques  (Ultimate) 

The  third  camouflage  condition  was  obtained  by  using  the  U.S.  Army 
Standard  Lightweight  Camouflage  Screening  System  deployed  over  the  vehicles. 

The  screens  were  erected  in  accordance  with  TM5-1060-200-10^/ .  Camouflage  im¬ 
provements  were  made  to  the  screens  by  tying  natural  foliage  to  the  support 
systems,  around  the  screen  edges,  and  protruding  through  the  screens.  Oak  leaf 
mulch  was  dispersed  around  the  edge  of  the  camouflage  screens  to  break  ufp  the 
straight  line  edge-  effect.  Palmetto  fronds  and  aawgraas  clumps  were  placed 
over  the  batten  spreaders  of  the  support  system  to  reduce  shine  and  were  woven 
into  the  camouflage  screens  to  simulate  the  appearance  of  natural  foliage. 

4.0  TE8T  IMAGERY 

The  site  was  photographed  using  9  inch  strip  color,  aerial  film  at 
scales  of  1:5,000  and  1:10,0003/  each  with  60%  forward  overlap.  The  target 
location  was  identical  for  each  camouflage  condition.  The  end  product  was 
three  strips  of  imagery  at  each  of  the  scales  of  1:10,000  and  1:5,000.  The 
1:10,000  scale  of  imagery  was  17  frames  long  while  the  1:5,000  scale  imagery 
was  5  frames  in  length. 

5.0  TEST  PROCEDURES 

The  subjects  consisted  of  99  pairs  of  operational  Image  Interpreters 
(Il’s).  They  were  instructed  on  the  purpose  and  tasks  to  be  performed.  Each 
team  had  three-quarters  of  an  hour  to  detect  targets  on  one  of  the  strips  of 
imagery  scaled  1:10,000.  At  the  end  of  this  time  period,  the  II ’s  were  given 
the  corresponding  strip  of  imagery  scaled  1:5,000  and  a  set  of  equipment  keys 
that  they  studied  in  an  attempt  to  identify  the  two  targets  of  interest.  They 
were  allowed  15  minutes  to  determine  an  identification.  No  team  of  II 's 
viewed  more  than  one  camouflage  condition. 

6.0  RESULTS 

The  percentages  of  detection  for  each  of  the  two  test  items  were  deter¬ 
mined  for  each  of  the  three  camouflage  conditions.  A  statistical^/  analysis 
of  the  data  revealed  that  of  the  two  items  investigated,  the  item  embedded  in 
the  trees  indicated  no  significant  differences  between  the  percentages  of  detec¬ 
tion  for  the  three  conditions  of  camouflage  identified  in  Section  3.0.  The 
item  not  as  deeply  embedded  in  the  trees  was  identified  by  more  II's  and  yielded 
significant  differences  between  camouflage  conditions  as  follows: 

o  The  pattern  painted  item  was  detected  significantly  (  a-  0.025) 
more  often  than  Che  pattern  item  with  standard  camouflage  screens,  (Ultimate 
Condition) 


o  The  pattern  painted  item  with  prototype  camouflage  was  detected 
significantly  (  a ■  0.025)  more  often  than  the  pattern  painted  item  with  stan¬ 
dard  camouflage  screens.  (Ultimate  Condition) 
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The  percentages  of  identification  for  each  of  the  two  teat  items  were 
determined  for  each  of  the  three  conditions.  A  statistical  analysis  of  the 
data  revealed  that  both  target  items  yielded  significant  differences  between 
camouflage  conditions  as  follows: 

o  The  pattern  painted  items  were  identified  significantly  (  ot  ■  0.025) 
more  often  than  the  pattern  painted  items  with  the  addition  of  prototype  camou¬ 
flage. 


o  The  pattern  painted  items  were  identified  significantly  ( a »  0.025) 
more  often  than  the  pattern  painted  items  with  the  addition  of  camouflage 
screens  and  techniques.  (Ultimate  Conditions) 

7.0  DISCUSSION 

The  results  of  the  study  indicated  that  the  design  of  the  experiment  was 
successful  in  statistically  evaluating  the  base  line  camouflage  condition  against 
both  the  prototype  camouflage  condition  and  the  ultimate  camouflage  condition. 

A  look  at  the  detection  data  indicates  that  for  the  item  not  embedded  in  the 
trees,  the  prototype  camouflage  yielded  significantly  (<*  ■  0.025)  leaa  detec¬ 
tions  then  the  bese  line  camouflage  condition.  However,  it  wee  detected  signi¬ 
ficantly  more  than  the  ultimate  camouflage  condition.  This  finding  telle  the 
camoufleur,  that  while  the  prototype  camouflage  for  the  item  has  decreased  de¬ 
tections,  more  development  is  required  to  bring  it  up  to  speed  with  the  ultimate 
camouflage  condition.  However,  the  trade  off  of  the  amount  of  time  and  manpower 
required  to  decrease  the  number  of  detections  must  be  considered.  The  number 
of  detections  for  the  item  embedded  in  the  trees  is  ao  lew  that  no  further  develop¬ 
ment  is  necessary. 

The  experimental  design  was  also  successful  in  statistically  evaluating 
the  base  line  camouflage  condition  egeinet  the  prototype  end  ultimate  camou¬ 
flage  conditions  for  the  teak  of  item  identification.  In  this  study,  both  the 
prototype  end  ultimate  camouflage  conditions  yielded  significantly  ( a  ■  0,025} 
less  identifications  than  the  base  line  condition.  There  was  no  significant, 
diffarance  between  the  number  of  identifications  for  the  prototype  and  ultimate 
camouflage  conditions.  This  fact  would  indicate  to  the  camoufleur  that  no  ad¬ 
ditional  refinement  is  required  to  reduce  rate  of  identification. 

8.0  SUMMARY 

The  purpose  of  this  study  wee  to  design  the  field  evaluation  of  a  camou¬ 
flage  ayetem  in  such  a  manner  that  the  camoufleur  could  statistically  differen¬ 
tiate  between  the  base  line,  prototype,  end  ultimate  levels  of  camouflage  for 
both  rates  of  detection  and  identification.  With  this  information,  the  camou¬ 
fleur  could  determine  the  feasibility  of  additional  prototype  development  to 
approach  the  effectiveness  of  the  ultimate  camouflage  condition.  The  ultimate 
camouflage  condition  was  defined  aa  having  no  restrictions  on  time  or  manpower 
in  its  erection  or  retraction  using  standard  materials  and  method*.  This 
study  was  conducted  in  Avon  Perk,  Florida.  The  result!  from  the  data,  using 
operational  II '•  showed  that  the  desired  experimental  discrimination  between 
camouflage  levels  was  achieved. 
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ESTIMATING  MEAN  LIFE  FROM  LIMITED  TESTING 
Donald  W.  Rankin 

Army  Materiel  Test  and  Evaluation  Directorate 
US  Army  White  Sands  Missile  Range 
White  Sands  Missile  Range,  New  Mexico  88002 


Exact  probability  formulae  are  developed,  with  no  restrictive 
assumptions,  for  use  with  tests  which  produce  data  of  the  constant  failure 
rate  type.  Although  universally  valid,  the  formulae  are  particularly  apropos 
when  straitened  test  circumstances  are  dictated.  Programming  suggestions  are 
Included.  (*- — - 

1.  INTRODUCTION.  This  paper  Is  a  sequel  to  the  one  entitled  Estimating 
Reliability  from  Small  Samples  and  presented  before  the  twenty-second 
conference  on  the  Design  of  Experiments  In  October  1976  [4]. 

The  Poisson  distribution  Is  treated  In  a  manner  parallel  to  that  afforded 
the  binomial  distribution  In  the  earlier  paper. 

2.  DEFINITION  OF  EVENT.  Probability  statistics  require  the 
Identification  of  a  unit  commonly  called  event  or  trial.  Often  this 
Identification  Is  self-evident.  Suppose  a  test  consists  of  drawing  a  sample 
of  specified  size  (n,  say}  from  a  larger  population  of  similar  Items,  then 
determining  the  number  of  defective  Items  (k)  In  the  sample.  It  requires  no 
stretch  of  the  Imagination  to  say  that  drawing  that  sample  of  size  n 
constitutes  an  event  or  trial  and  that  the  failure  ratio  k/n  Is  the  reeu.lt  of 
that  event.  It  Is  to  be  noted  that  the  failure  ratio  is  dimensionless;  l.e., 
k  and  n  are  measured  In  the  same  units. 

Identification  Is  not  always  so  clear-cut.  For  example,  suppose  an 
operator  of  heavy  trucks  notices  that  In  the  preceding  six  months,  he  has 
experienced  13  major  mechanical  breakdowns--one  every  two  weeks,  on  the 
average.  The  definition  of  failure  Is  obvious,  but  what  Is  a  euooeeel  To 
what  do  we  add  k  to  get  n,  the  sample  size?  The  mathematical  answer  Is  that  n 
+  •.  But  this  Is  also  a  useless  answer;  no  realistic  test  design  could 
require  an  infinite  sample  size. 

To  avoid  facing  this  dilemma,  let  us  arbitrarily  define  event  In  some 
convenient  unit  different  from  that  In  which  k  Is  expressed.  As  a 
consequence,  we  no  longer  have  a  failure  ratio.  In  Its  place  we  substitute  a 
failure  rate--of  k  per  event.  Thus  the  failure  rate  depends  upon  an  observed 
k,  but  upon  a  defined  event. 
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To  return  to  the  truck  operator,  let  us  say  that  examination  of  the  log 
>ooks  reveals  a  total  operating  mileage  of  267150  for  the  period  In  question. 
[This  figure  (267150  miles)  Is  taken  as  the  definition  of  event.  The  observed 
'failure  rate  then  becomes 


13  failures 
7E7T5U"mTTeT 


0.0000486667  failures  per  mile. 


It  Is  sometimes  reqarded  as  preferable  to  express  the  reciprocal  of  the 
failure  rate,  calling  It  mean  life.  Thus  we  would  have 


267150  miles 
13  f al lures 


■  20550  mean  miles  between  failures. 


The  term  event  can  be  defined  In  any  of  a  variety  of  units— area,  volume, 
weight,  time- -almost  anything  that  can  be  measured. 

3.  POISSON  PROBABILITY.  Consider  the  well-known  series 


x 


I  4r“»  (k  “  °»  2»  3* 

k-o 


.  .) 


(1) 


This  series  converges  for  all  finite  values  of  x,  provided  only  that  x  remains 
constant.  Multiplying  by  e“*  produces 


i  .  T  alii 

k«o  l!i 


(2) 


Poisson  noted  (1837)  that  If  x  Is  a  constant  failure  rate  and  k  Is  a  non¬ 
negative  Integer,  the  probability  of  observing  exactly  k  failures  during  an 
event  Is  given  by  the  appropriate  term  of  the  above  expansion;  l.e.,  by 


P(k)  .  aLd 

Ik 


This  last  expression,  then  Is  a  probability  function  In  the  discrete  variable 
k.  Unfortunately,  however,  It  does  not  suffice.  In  most  test  designs,  It 
will  be  possible  to  define  event  arbitrarily  and  to  observe  the  value  of  k 
exactly,  but  nothing  will  be  known  about  x.  Usually,  In  fact,  x  will  be  the 
principal  value  sought.  A  probability  function  In  x  Is  required. 
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It  is  a  continuous 


Now  x  can  take  on  any  non-nepative  value;  l.e., 
variable  within  the  limits  0  <  x  <  •.  Necessarily 

1  *  /  f(x)  dx 
;o 

defines  f(x)  as  the  required  probability  function  in  x,  whatever  form  it  may 
take.  With  k  fixed,  the  expression 


xk  e'x 
Ik 

becomes  a  density  function  In  x  (though  not  necessarily  a  probability  func¬ 
tion).  It  Is  necessary  to  evaluate  the  definite  integral 


I 

k 


/ 


o 


k  -x 
x*  e  * 


Ik 


dx 


Since  k  Is  constant,  can  be  taken  outside  the  integral  sign,  leaving 


{ k  I  ■  /  xk  e"x  dx  *  r  (k  +  1) 
—  k  o 


but  also,  k  Is  an  Integer,  hence  |k.  ■  r  (k  +  1). 
It  Is  seen  that  Ik»  1,  and  therefore  that 


is  the  required  probability  function  In  the  continuous  variable  x. 

It  Is  helpful  to  Inspect  graphs  of  the  probability  function 

f(*)  -  *yk£'X  (4) 

Several  are  depicted  In  Figure  1  for  various  Integer  values  of  k.  Among 
features  which  should  be  noted  are  the  following: 

1)  When  k  ■  0,  the  function  degenerates  to 

f (x )  ■  e"x  (5) 


and  Is  most  easily  treated  as  a  separate  case. 
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2)  f'(x)  «  {kxk_1  e'x  -  e“x  xk}  =  -k  —  ?  *  (k  -  x) 

I  k  I  k 

Thus  a  maximum  occurs  when  x  *  k. 

3)  f"(x)  =  xk'2  £‘X  *  {k(k-l)  -  Zkx  +  X2}  (7) 

I  * 


A  point  of  Inflection  Is  found  whenever 

x2  -  2kx  +  k(k  -  1)  *  0, 


l.e.,  when  x  ■  k  ±  /IT.  For  some  programming  purposes,  when  k  *  1,  the  origin 
may  serve  as  the  mlsslnq  point  of  inflection.  The  slope  there  Is  unity. 

4)  Every  curve  crosses  every  other  curve  exactly  once,  and  In 
consecutive  order. 


5)  Two  consecutive  curves  Intersect  at  the  maximum  point  of  the 
second,  since  the  only  non-trlvial  solution  of 


occurs  when  x  ■  k  +  1. 


4.  TRANSFORMING  THE  PROBABILITY  FUNCTION. 


If  the 
Itself. 


case  k  ■  0  Is  treated 
Letting  x  ■  kw, 


m 

f 

kw»o 


since  merely  employing  the  transformation  will  not  affect  the  value  of  this 
definite  Integral.  But  the  probability  function  In  w  Is 


Basically,  this  transformation  rescales  the  abscissae  by  1/k,  and  hence 
the  densities  (ordinates)  by  k,  thereby  preserving  area.  Several  graphs  of 
this  function  are  shown  In  Figure  2,  Notice  that  every  curve  has  Its  maximum 

point  at  w  «  1.  Also,  w  +  1.  Points  of  inflection  occur  at  1  ±  1//T. 
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Although  the  transformation  Is  useful  for  studying  this  family  of 
functions,  It  matters  very  little  whether  levels  of  confidence  are  computed 
from 

z  z/k 

/  f ( x )  dx  or  J  g(w)  dw. 
x°o  w-o 

In  this  paper,  the  form  In  f(x)  will  be  used. 


5.  INTEGRATION  BY  PARTS.  When  a  function  Is  defined  by  (or  can  be 
described  as)  a  definite  Inteqral,  very  frequently  It  will  be  found  that 
repeated  Integration  by  parts  will  produce  an  expansion  suitable  for 
computing.  In  fact,  as  In  the  Instance  at  hand,  It  may  be  possible  to  expand 
In  either  ascending  or  descending  factorials  (or  powers,  as  the  case  may  be), 
thereby  producing  two  different  expansions,  both  of  vrtilch  are  valid.  Usually, 
one  will  appear  In  the  familiar  form  of  a  power  series  which  converges  more 
rapidly  for  smaller  values  of  the  argument.  The  other  will  be  the  associated 
asymptotic  expansion.  If  the  parameter  which  appears  In  the  factorial  part  of 
the  probability  function  can  be  restricted  to  Integer  values  only,  the  asymp¬ 
totic  expansion  becomes  finite  In  length  and  Is  an  exact  expression. 

The  sought  probability  Integral  can  be  stated 


2  2  k  -x 

P(z)  ■  /  t'(x)  dx  ■  /  *7-® —  dx 

x-o  x-o 


(9) 


and  gives  the  probability  that  x  does  not  exceed  the  (perhaps  arbitrary)  value 
z. 


Can  the  Indefinite  Integral 
fixed,  positive  Integer? 


/ 


sLsc! 

Ik 


dx  be  evaluated  by  parts,  k  being 


a 


x  xk 

Let  u  *  e  and  dv  ■  dx. 


Ik 


Then  du 


-e*xdx  and  v 


-k+1 

i ikTT 


Thus  / 


yk  p-X 

*  e  dx 
Ik 


p-x  *k+l  ,  p-x  *k+l 
-2 — * —  +  f  £ — * —  dx 


I  k+1 


I  k+1 


It  Is  apparent  at  once  that  the  second  integral  Is  like  the  first,  save  k 
has  been  augmented  by  unity.  It  Is  clear  that  the  process  can  be  reapplied 
endlessly,  yielding 


/  dx 

1  Ik 


1 ! 

1-1 


e~x  X*  ♦  1 
I  k  +  1 


(1  «  1,  2,  3, 


)• 
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Pass  1  iq  to  the  lower  limit  of  the  definite  Integral  (x  =  0),  the  sum  vanishes, 
since  x  factors  every  term.  (It  may  be  more  correct  to  say  that  the  sum 
reduces  to  the  constant  of  Integration.)  Thus 


P(z) 


X 


1 


! 

X-0 


A 


-X 


dx 


e"2  zk+i 
|k_+_5 


(10) 


The  term-to-term  recurrence  ratio  Is  z/(k  +1).  Since  z  is  constant  while 
(k  >  1)  Increases  without  bound,  the  series  will  (eventually)  converge  for  all 
positive  values  of  z. 


Now  let  u  ■  and  dv  «  e"xdx, 
Ik 


Then 


du 


dx  and  v  *  -e'x, 


whence 


dx  ■ 


xk~*  e'x 
Ik  -  1 


dx . 


Noting  that  -e"x  will  factor  every  term,  we  can  write  the  result  In  the  form 


yk  o“X  _y  yk  yk“l  y2  .1 

/  * — fi —  dx  ■  -e  x  +  & - +...+■*-  +  x  +  l1 

J  Ik  |k  |k  -  1  2 


At  the  lower  limit  (x  "0),  the  right-hand  member  becomes 

The  definite  Integral  thus  Is  given  by 


P(z) 


yk  p-X 

dx 

x*o  |k 


1 


1  -  e'z  ^  + 


.k-1 


Ik  Ik  -  1 


•  •  • 


z  +  1 


Are  the  two  solutions  equivalent?  Is  It  true  that 


.67? 


i-i  ILU. 


1  -  e 


-z 


il  +  z  +  +  .  .  .  + 


rk-1 


Ik  -  1  Ik 


(ID 
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Multiplying  by  ez  and  transposing,  it  is  seen  that 


4 


1  +  z  + 


7k )  »  7k+1 

+  M  +  T  1 - 

Iki  iSi  Ik  +  1 


is  the  well-known  Maclaurin  series  for  ez,  Therefore  the  two  solutions  are 
Indeed  equivalent. 

It  is  a  fact  that  if  the  upper  limit  of  Integration  be  taken  at  the 
maximum  (w  ■  1;  l.e.,  z  ■  k),  the  level  of  confidence  will  always  be  less  than 
1/2  and  hence  of  little  statistical  Interest.  (See  Table  1 . ,  However,  the 
araument  z  «  k  has  an  important  use  of  a  different  sort.  It  enables  us  to 
select  a  series  for  computlnq  whose  terms  are  known  to  decrease  monotonic  ally. 
This  results  in  worthwhile  economy  for  larqer  values  of  k.  There  are  two 
cases  to  consider. 


First:  Let  0  <  z  <  k.  The  series 


e*zzk'f1 
|k  +  1 


is  chosen  for  use.  Obviously,  the  term-to-term  recurrence  ratio  is  given  by 
z/(k  +1).  Under  the  stated  conditions,  this  is  always  less  than  unity. 


Second:  Let  z  >  k.  The  formula 


P(z)  «  1  -  e 


,k-l 

+  * - + 

Ik  -  1 


♦  +  z  ♦ 


is  used.  The  recurrence  ratio  is 


&  *  *  ~  i,  (1  -  1,  2,  3 . k) 

z 

which  aqaln  Is  less  than  unity.  For  large  values  of  k,  the  interior  series 
can  be  summed  as  though  it  were  an  infinite  series,  thus  achieving  a  laudable 
saving  in  the  number  of  terms  required. 

6.  COMPUTING  A  LEVEL  OF  CONFIDENCE  (z  >  k).  The  value  of  z  may  be 
derived  from  any  source,  or  It  may  be  arbitrarily  specified.  The  proper 
formula,  as  we  have  seen,  is 


P(z)  »  1  -  e-z  —  + 

Ik  |k  -  1 


.  .  +  z  +  I 
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TABLE  1 


CONFIDENCE  LEVEL 


k 


0 

1 

2 

J 

4 


AT  MAXIMUM  ORDINATE 


/  f(x)  dx 


0.000000 

0.264241 

0.323324 

0.362768 

0.371163 


5 

6 

7 

8 
9 


0.384039 

0.393697 

0.401286 

0.407453 

0.412592 


10 

12 

15 

20 

30 


0.416960 

0.424035 

0.431910 

0.440907 

0.451648 


50 

100 

200 

400 

1000 


0.462483 

0.473438 

0.481206 

0.486706 

0.491591 


When  k  Is  small  (k  <  12,  say),  the  resulting  finite  expression  submits 
easily  to  direct  computation.  But  when  k  Is  very  large,  two  difficulties 
arise. 


First:  The  number  of  terms  becomes  excessive.  If  the  series  Is 
summed  as  though  it  were  an  infinite  ser1es--i .e. ,  the  relative  size  of  each 
new  term  is  observed--the  process  can  be  truncated  when  additional  terms  no 
longer  affect  the  result  In  the  computer. 

Second:  Large  factorials  will  overflow  the  computer.  To  circumvent 
this,  the  first  term  of  the  series  is  computed  by  logarithms.  Stirling's 
formula  (k  >  11)  is  given  by 

In  |k  -  0.91893  85332  +  (k  +  4)  In  k 
e  “  2  e 


-  k  + 


12k 


1  - 


30k  * 


(12) 


The  first  term  Is  (disregarding  sign) 


;  hence,  Its  logarithm  will  be 


k  lnez  -  z  -  lneU,  which  should  not  cause  overflow  within  the  range  of  useful 


numbers. 


7.  COMPUTING  z  WHEN  A  LEVEL  OF  CONFIDENCE  IS  SPECIFIED.  (L  -  P(z)  >  0.5) 
No  new  formula  Is  available  for  the  Inverse.  Instead,  successive  approximations 
z0,  zi,  Z2,  .  .  .  are  computed  until  a  steady  state  Is  reached.  Newton's 
method  serves  very  well.  See  the  discussion  In  [4]  pp.  279-280. 

For  any  z<,  compute  P(z<),  f ( )  and  f'(z^).  The  required  Incremen¬ 
tal  area  Is  of  course  P(z)  -  P(z^).  We  approximate  this  area  with  a 
trapezoid  of  width  Az  whose  ordinates  are  f(z^)  and  f(z^)  +  Azf'(zi). 

We  have  seen  earlier  that  the  first  term  of  the  wanted  series  for  r’(z^)  Is 


e _ 1 

Ik 


Also  f(z^) 


and  f'(z^)  -  -  1)  f(z^  . 
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The  approximating  trapezoid  Is  given  by 

P(z)  -  P (zj)  «  az  {f (z1 )  +  f-(2i )j 

which  can  be  solved  for  az. 


Az  «  -f(Zj)  ± 


£ 


[HiQr  2f'(zQ  [P( 2)  -  P(zQ1 


f'(Zl) 


Since  ultimately  az  +  o,  It  Is  apparent  that  the  positive  square  root  yields 
the  true  solution.  Noting  that 


IILlL.  Li 
f'U()  k  -  2, 

the  formula  can  be  simplified  to 


az 


f'Ui) 


U3) 


The  process  Is  stable  when  started  from  the  right-hand  point  of  Inflection; 
l.e., 


Z0  -  k  +  JT  (14) 

8.  THE  BEST  ESTIMATE  OF  THE  FAILURE  RATE.*  For  a  specified  level  of 
confidence  L,  the  general "soTu'iTon  "o‘f"tW"pfob'ab  1 1 1  ty  Integral  Is 

s 

L  ■  /  f(x)  dx. 


There  are,  of  course,  an  unlimited  number  of  solution  oalrs  (a,  s)  which 
satisfy  this  equation.  Up  to  this  point,  we  have  concerned  ourselves  with  the 
case  a  *  o.  This  form  properly  is  used  to  test  for  compliance  with  an  Imposed 
standard, 


Sometimes,  however,  that  standard  is  absent,  unrealistic,  or  even 
erroneous.  Put  It  Is  still  required  to  make  a  meaningful  statement  about  the 
failure  rate.  In  this  situation,  the  Beat  Betimate  1$  recommended. 
Essentially,  that  solution  pair  (a,  s)  Is  chosen  which  minimizes  the 
difference  |  s  -  a  |. 


♦See- [4]  pp.  267-270. 
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Values  of  a  and  s  thus  determined  are  designated  by  a  tilde  (a,  s). 

Some  properties  of  the  Best  Estimate  of  the  Failure  Rate  are: 

a.  s  *  a  is  minimum,  by  definition. 

b.  The  limits  of  integration  lie  on  opposite  sides  of  the  maximum; 
i  .e. ,  a  <  k  <  s. 

c.  The  ordinates  at  a  and  5  are  equal;  i.e,  f ( a)  »  f(s). 

d.  The  solution  is  unique. 

There  are  several  steps  in  the  solution. 

Step  One.  For  any  s^,  compute  f(s^),  f>(si),  P(s1). 

(To  begin,  set  sQ  ■  k  +  /k. ) 

Step  Two.  For  each  s<k  solve  for  the  value  a  <  k  such  that  f(a)  ■  f(s^). 
For  any  aj,  compute  f ( a ^ )  and  f'Uj). 

Then 


Aa 


(15) 


The  process  Is  repeated  until  f(a)  and  hence  a  Is  found  to  the  desired 
accuracy.  This  value  of  f ( a)  Is  then  associated  with  f(s^)  by  appending  the 
subscript  i  .  (The  subscript^  Is  dropped,  being  no  longer  necessary.)  For 
every  new  value  of  s^ ,  the  a-process  Is  begun  afresh  by  setting 

aJ  "  k  -  /iT. 

Step  Three.  The  value  f(aj  -  f(s<)  having  been  found,  compute 
P(at).  (The  values  for  a^  and  f'fa^)  will  already  have  been  computed.) 

The  desired  Incremental  area  Is  L  -  P(s^)  +  P(a^) 


Step  Four.  The  Incremental  area  always  will  appear  in  two  separate 
parts.  The  ratio  of  these  areas  can  be  estimated  quite  closely  by  the  slopes. 
Thus 


\)  +  P(«i) 


’)(- 


f'Ui)  -  f'( 


will  appear  on  the  right.  It  Is  convenient  to  express  the  ratio  In  terms  of 
the  ordinates. 


i  — j  i  nrinfi  II  ffi  T*  1 1  f  1 1  f  i 


188 


f'Uf)  -  (S1  > 


But  since  ffaO  *  f(s^),  this  value  can  be  cancelled  from  numerator  and 
denominator,  leaving 


s^k  -  a1 ) 

-  f " ( s  1 )  k(s1  -  a^' 

Thus  a  suitable  approximating  trapezoid  Is  given  by 


{ ■  <>(*1) +  ■ 4S  if(si>  *  r  f'(s,)>  (17) 

which  can  be  solved  for  as  by  the  method  of  Section  7,  above. 

9.  EXPRESSING  RESULTS  IN  TERMS  OF  MEAN  LIFE.  It  should  be  noted  that 
the  methods  developed  In  this  paper  are  virtually  Independent  of  the 
definition  of  Event.  (Event  often  will  be  synonymous  with  Duration  of  Test.) 
Suitable  values  of  a  and  s  (or  z,  as  the  case  may  be)  having  been  found,  It  Is 
apparent  that  they  should  be  expressed  In  the  units  failures  per  event.  If  at 
this  point  the  definition  of  event  is  Imposed,  the  results  can  be  expressed  In 
failures  per  mile  or  failures  per  hour  or  whatever. 

Now  the  simple  reciprocal  converts  to  mean  life,  it  should  be  remembered 
that  taking  the  reciprocal  reverses  the  sense  of  inequality  signs. 
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APPENDIX  A 


CHI-SQUARE  AND  OTHER  POISSON-RELATED  FUNCTIONS 


Let  us  define  the  following  special  functions: 
Incomplete  exponential  function: 


e  (x)  -  1  +  x  +  +  .  .  .  +  A 

n  |2  |n_ 

The  series  consists  of  n  +  1  terms. 

Gamma  function: 

*5  x- 1 

r(x)  ■  /  e  t  dt 
o 


Thus  r(x  +  1) 


m  -t  X 

7  e  t  dt. 

o 


(x  >  o) 


Incomplete  gamma  function: 


.  .  *  -t  x-1 

y(x,z)  -  /  e  t  dt 

and,  of  course,  o  <  z  <  •. 

Prym's  function: 

•  -t  x-1 

r(x,z)  *  /  e  t  dt 
z 


(X  >  0) 


(z  >  o) 


Immediately  It  Is  seen  that 

y(x,z)  +  r(x,z)  «  r(x) 

and  that  dividing  both  sides  of  this  equation  by  r(x)  will  produce  a 
probability  relationship. 


Thus  we  can  state 


P(x,z) 


-r(x,z) 

r(x) 


r(x,z) 

r(x)  * 
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Now  .for  ary  particular  problem,  x  and  hence  r(x)  will  remain  fixed.  In  terms 
of  Prym's  function  we  can  write 


n<  .  1  «  -t  x-1 

P{x,z)  »  1  -  — -  f  e  t  dt. 
r(x)  i 

It  Is  easy  to  develop  r{x,z),  using  repeated  integration  by  parts.*  It  is 

found  that 


-z  x-1  *  (x  -  1)  (x  -  2)  ...  x  -  s 
r(x,z)  ~  e  z  l  - 

s«o  zs 

Is  a  valid  asymptotic  expansion  for  fixed  x  and  large  z. 

When  x  Is  an  Integer,  the  series  terminates. 

When  x  Is  not  an  Integer,  the  terms  of  the  series  alternate  In  sign  after 
s  >  x.  The  series  diverges  after  s  >  x  +  z. 

Let  us  replace  x  with  k  In  the  formulae  In  order  that  x  can  be  employed 
as  a  variable  of  Integration.  Thus  the  formulae  restated  appear  as  follows: 

P(k,2)  .  1  -  liKiU.  ■  1  -  -ij-  [e,.!  (2)] 


P(k.z)  .  l  -  e'1  t1"1  V  (k  -  1)  (k  -  2)  ■  ■  ■  (k  -  s) 
r<k>  s-o  zs 

When  k  Is  a  positive  Integer. 

This  case  of  k  being  a  positive  Integer  was  studied  at  length  In  Its 
application  to  sampling  distributions  by  Helmert  (1876)  and  K.  Pearson  (1900). 
Thus  arose  the  statistics  of  the  x2  distribution.  The  exponent  2  In  x2  has 
little  significance  beyond  ensuring  that  the  parameter  Is  non-negative. 


The  x2  probability  function*  Is  defined  by: 


P(x2lv) 


2v/2  r( v/2)  0 


f  (t)(v/2)-l  e-t/2  dt 


q(x2|v)  ■  — — - J  (t)(w^2)"l  e*1*/2  dt 

2v/2  r( v/2)  x2 

P(x2lv)  +  Q(x2!v)  ■  1 


Comparing  this  to  the  earl ler-derlved 


P(k,z) 


'(k,z)  1 


r(k)  r(k) 


/  e"x  xk_1  dx, 
x«o 


It  Is  seen  that  the  only  differences  are  In  the  scaling  of  the  parameters. 
For  let  v  ■  2k.  Then 

,  1  I2  k-1  -t/2 

P(x2|2k)  ■  — - /  t  e  dt 

2kr(k)  o 


•so  ur  c--> 

Now  let  t  ■  2x,  from  which  dt  ■  2dx.  Replacing  the  variable  of  Integration, 


l  2x*x2  x-1  -x 

P(  X2 1 2k )  «  /  x  e  dx 

r(x)  >2xmo 

and  It  Is  seen  that  x2  ■  2z  properly  scales  the  limit  of  Integration. 

When  v  ■  2k  is  an  ODD  integer,  two  things  happen.  r(k)  contains  the 
,  ?  (k  -  1)  (k  -  2)  .  .  .  (k  -  s) 

factor  /?  and  J.  - - —  does  not  terminate.  The 

s-o  zs 

behavior  (accuracy)  of  the  asymptotic  expansion  near  z  «  k  -  s  must  be 
Investigated. 


*$ee  [1]  26,4  page  940. 
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APPENDIX  B 


PROGRAM  PLANNING  -  POISSON 


1.  INTRODUCTION.  As  a  general  rule,  the  only  variable  of  observation 
will  oe  k,  tne  number of  failures.  The  variable  of  Integration  will  be  x, 
with  z  one  of  Its  extreme  values  (limits  of  integration). 

It  is  necessary  to  define  event  In  some  suitable  unit  (time,  distance, 
mass,  volume,  etc.);  e.g.i'auant  ■  4240  hours.  Event  often  Is  synonymous  with 
Duration  of  Teet . 

Many  formulae  of  interest  are  greatly  simplified  If  expressed  as 
functions  of  f(x)  or  of  f(z).  Thus 


f(x) 


-x  k 
e  x 

Ik- 


f-(x)  -  -  1^  f(x) 

fM(x)  -  ^2-  ij  f { x ) 


*  -*  k+1 

p(z)  -  T  * — * — 

w  IL1 1 


(1  »  1,  2,  3,  .  .  .) 


Ti  +  T2  +  T3  +  ‘  *  T1  *  *  *  ’ 


T  ■  — 2 —  f(z)  and 
a  k  t-  1 


T  •  — S —  T 

j  k  +  J  J-l 


Also, 


1  -  Pf z )  -  T  +  T  +  T  +  ...+T  +  .  .  . 
o  1  2  1 


T  -  f(z) 
o 


T  -  ^-1  T 

j+ 1  2  i 

This  latter  series  terminates  when  k  *  „1. 

For  large  values  of  z,  compute  f(z)  by  logarithms. 

1ne  f(z)  ■  k  lnez  -  z  -  lne|k. 

Stirling's  formula  for  IngU  Is  useful  here.  If  k  does  not  change,  It  need  be 
computed  but  once. 

2.  COMPUTING  l  (z  specified).  Equations  (4),  (5),  (9),  (10)  and  (11). 


Enter  data 


subroutine  for  zero  failures 


Compute  In  |k.  (If  k  >  15,  use  Stlrllnq's  formula.) 
e  •“ 

Compute  k  In  z  -  z. 
e 

k  -z 

Compute  f{?)  ■  -2 — § —  -  T 
|k  o 


Compute  L  •  /  f (x )  dx  from  one  of  the  methods  In  the  previous  paragraph. 


(if  k  >  15 


and  z  <  k,  use  L 


-z  W\ 

T  £ - 2 - j 

1-1  IJL til 
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•  Si 


I 


m 


KT?* 


Ki 


I 


ft 


c<- 


§ 

sV 

Si1 

ta 


(13). 


3.  COMPUTING  z  (L  specified).  To  the  above,  add  equations  (6),  (7)  and 


Enter  data 


subroutine  for  zero  failures 


Subsequent  portion  of  method  assumes  L  > 


Assign  z  ■  k  +  A T* 


Label 


0 


Apply  method  of  paragraph  2  above  to  compute  L  ■  /  f(x)  dx 

o  o 


Compute  f'(z0)  ■  -  1^  f(zQ)  and 


AZ  - 


1-k 


tf/*\  \2  20~ 

+  — 


-  h) 


(*i> 


end , 


Assign  z  ■  z^+  az 


Return  to  Label 


0- 
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4.  COMPUTING  BEST  ESTIMATE  OF  THE  FAILURE  RATF  (L  Specified 
Enter  data 


*  Employ  paragraph  3,  above. 


Assign  s  ■  k  +  /I <T 
o 


Label 


0 


Compute  f  f(x)  dx  by  method  of  paragraph  2  above. 

»  A 


Compute  f'(s^)  ■  -  1^  f (s^ ) 


-+  asslan  a  »  k  -  /IT* 
o 


Label  |T| 

Compute  f(aj)  and  f " ( aj ) 

f(Sj)  -  f(8j) 


Compute  Aa  " 


f ' ( M ) 


Compute  and  store  a,  ,  ■  a..  +  Aa 


a 


Compute  /  f(x)  dx  by  methods  of  paragraph  2  above, 
'o 

(f  a  .  -a  k+l\ 

r k  >  15> ,(x)  dx  ■  ill  Vir) 

The  needed  Increment  of  area  Is 


A  -  L  -  r1  f(x)  dx  +  f1  f (x )  dx 

»  A  'A 


The  approximating  trapezoid  yields  (momentarily  dropping  subscripts  for 
convenience) 
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An  Bffioiant  Method  for  Determining  the  "A*  end  "B"  Design  Allowables 


DONALD  DIAL  AND  LUCIANO  8PIR1DGLI0ZZI 
Army  Msterisls  end  Mechanics  Research  Center 
Watertown*  Massachusetts,  (J8A 


ABSTRACT 


Suggested  statistical  procedures  for  obtaining  material  "A"  end  "B*  allow¬ 
ables  from  both  oomplete  and  censored  samples  are  outlined  in  this  paper.  The 
sllowsbles  represent  a  value  determined  from  a  specified  probability  of  sur¬ 
vival  with  a  93%  confidence  in  the  assertion.  The  survival  probabilities  are 
.99  for  the  *A*  allowables  and  .90  for  the  "B”  allowables.  Both  parametric 
and  non-parametrio  statistical  models  are  evaluated  with  respeot  to  their 
desirability  in  obtaining  the  allowables.  Exploratory  data  analysis  proce¬ 
dures  are  introduced  in  order  to  determine  acceptable  distribution  functions 
for  representing  the  data  in  addition  to  recognising  outliers  (bad  data)  or 
multi-modality.  Zt  is  demonstrated  from  a  variety  of  materials  test  data  that 
allowable  determinations  require  prior  application  of  exploratory  data  analy¬ 
sis  procedures  in  order  to  assure  acceptable  results.  The  analysis  also 
provides  a  process  for  recognising  either  poor  testing  procedures  or  inferior 
material  processing. 

The  two  parameter  Meibull*  normal*  lognormal  distribution  functions  are 
the  proposed  statistical  models  for  computing  the  allowables  (whan  non- 
parametrio  methods  ere  not  applicable) .  They  will  usually  provide  an  accept¬ 
able  range  of  possible  allowable  values.  The  Informative  Quantile  Punotion  is 
applied  to  the  test  data  in  order  to  seleot  the  function  that  best  represents 
the  data.  In  determining  the  allowables*  the  desirability  of  the  Weioull 
function  application  is  shown  when  limited  number  of  probability  ranked  data 
values  are  available  in  the  primary  region  (lower  ranked  numbers)  of 
interest.  The  required  conservatism  in  this  region  is  satisfied  while  alao 
satisfying  criteria  for  acceptability  of  the  data  representation.  The  exist¬ 
ence  of  multi-modality  or  gross  outliers  in  the  date  set*  will  in  some 
instances  introduce  excessively  conservative  estimates  of  the  allowables  when 
the  Meibull  function  is  applied.  If  the  multi-modality  case  is  a  reality  then 
a  suggested  procedures  using  the  Penalised  Likelihood  Method  is  used  in  con¬ 
junction  with  Cramer  Kao  lower  bound  estimate  for  the  95%  confidence  values. 

lx tensive  tables  for  samples  sises  (3(1)100)  have  been  developed  for 
computing  the  allowables  using  the  Meibull  function.  Use  of  the  Monte  Carlo 
Method  in  conjunction  with  maximum  likelihood  (ML)  relations  describes  the 
prooedure  used  in  obtaining  the  necessary  values  in  determining  the  allow¬ 
ables.  A  simple  computer  oode  has  been  made  available  so  that  ML  estimate  of 
Meibull  parameter  can  be  determined  thereby  resulting  in  direct  computation  of 
the  allowables. 

In  order  to  demonstrate  the  desirability  of  the  method*  allowables  have 
been  determined  for  Kevlar*  Graphite*  and  Class  composite  materials  subjected 
to  sheer*  tensile*  and  compressive  loads.  Most  of  the  test  data  was  obtained 
from  the  MIL-HDBK-17  (UBA  Army  Materials  and  Mechanics  Research  Center) 
project  for  composite  material  applications  in  aircraft  structures. 
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The  work  described  in  this  report  is  part  of  a  continuing  effort  to 
provide  atatietioel  prooedurea  for  determining  the  "A*  and  *B*  allowables  for 
the  M2L-HDBK-17  (Handbook  for  Ccaposlte  Material  in  Aircraft  Application) . 

The  preparation  of  thia  handbook  is  a  prime  mission  of  the  Army  Mater iala  and 
Mechanics  Research  Canter*  Reliability  Mechanics  and  Standardisation 
Division.  The  current  statistical  procedures  used  in  the  MXL-HDBK-5  is  not 
considered  applicable  in  the  determination  of  the  allowables  for  composite 
materials. 

The  sslaotion  of  an  adequate  statistical  model  (parametric  or  non- 
parametrio)  for  representing  material  strength  data  can  result  in  either  a 
conjectural  approach  or  s  costly  teat  program.  Often  times  the  normal  distrl" 
bution  will  be  seleoted  since  the  allowables  can  be  readily  determined  from 
tables.  Most  conventional  tests  for  determining  model  aooeptabllity  will 
rarely  rejeat  the  normal  funotion.  Unfortunately*  selection  of  the  normal 
distribution  oan  result  in  erroneous  allowable  estimates  due  to  extrapolation 
beyond  the  lower  ordered  test  results.  A  quote  from  Hahn  and  Shapito  (1] 
which  says*  'Although  many  models  might  appear  appropriate  within  the  range  of 
the  data*  there  might  well  be  in  error  in  range  for  whioh  prediotlona  are 
desired**  adequately  summarises  this  important  lsaue.  This  complicates  the 
issue  with  respect  to  allowable  computations*  therefore  possibly  requiring  an 
extreme  value  dlatributlon  representation  which  will  compensate  for  the  uncer- 
talnitles  in  the  lower  tall  region. 

The  non-par ametrio  method  is  not  realistic*  for  example*  if  an  "A*  allow¬ 
able  were  needed  for  a  apeoiflo  material*  the  300  testa  would  be  required. 

This  oould  result  in  an  extremely  costly  test  program*  in  that*  control  of 
environment  and  the  manufacturing  process  of  composite  material  must  be 
precise. 

Zn  order  to  address  the  allowable  computation  problem  more  rigoroualy*  the 
authors  have  examined  the  relative  merits  of  4  distinct  distribution  functions 
including  the  Penalised  Likelihood  Method  (3)  for  multi-modal  case.  Zn  most 
instances  the  2  parameter  Maibull  funotion  is  recommended.  This  extrema  value 
funotion  will  usually  provide  acceptable  estimates  of  the  allowables.  Either 
preolse*  or  slightly  conservative  estimates  will  be  obtained.  According  the 
Preudenthal  and  Qumbel  t3)*  the  use  of  the  Kelbull  distribution  to  represent 
the  distribution  of  the  breaking  strength  of  materials  has  been  justified  by 
using  extrema  value  theory.  In  order  to  recognise  the  most  desirable 
function*  the  authora  examined  the  Root  Mean  Bequence  error  (function  vs. 
ranked  test  data)  in  addition  to  the  Informative  Quantile  (IQ)  Function  (*) 
plots  of  the  data.  The  IQ  results  provided  an  excellent  description  of  test 
data  in  terms  of  a  specified  distribution  function. 

Zn  applying  the  Weibull  distribution  it  is  important  to  recognise  data 
with  outliers  in  vicinity  of  higher  ordered  values  in  addition  to  multi-modal 
behavior.  Data  contaminated  in  this  manner  will  usually  reduce  accuracy  in 
the  Meibull  allowable  computation.  The  Quantile  Box  Plot  13)  was  used  in 
determining  outliers  and  multi-modality.  This  method  proved  to  be  more 
reliable  then  the  conventional  robust  procedures  currently  being 

suggested  for  determining  outliers.  If  there  is  not  a  rationale  for  removing 
outliers  then  they  should  remain  in  the  data  set*  otherwise  erroneous 
estimates  of  the  allowablee  will  result.  In  the  multi-modal  case*  careful 
examination  of  test  procedures  and  material  processing  should  be  made  prior  to 
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acceptance  of  this  phenomena.  Zt  is  poaaibla  that  in  tastirtq  cartain  com- 
poaita  material!,  bi modal  bahavior  could  occur.  In  exploring  aoma  recent  taat 
results  from  Kevlar,  Graphite  and  Glass  oocpositea  in  addition  to  ceramic 
materials  resulted  in  occasional  bimodality  behavior.  In  most  cases  errors  in 
testing  or  materials  conditioning  and  processing  have  accounted  for  this 
situation. 

The  following  robust  method  for  applying  exploratory  procedural!  in  the 
examination  of  outliers  was  used  primary  as  verification  of  the  Quantile  Box 
Plot  results.  The  methods  singular  advantage  ia  that  visual  inspection  is  net 
necessary  in  recognition  o f\  the  outliers.  The  disadvantage  results  from 
arbitrariness  in  selection  of  scale  and  the  tuning  constant.  In  some 
instances  where  a  large  amount  of  skewnese  or  a  small  date  sets  exists,  then 
the  Quantile  Box  Plot  will  be  dispersion. 

teteiftniM.  Milted 

The  outliers  are  determined  in  a  formal  manner  by  applying  a  robust  method 
involving  application  of  the  MX.  estimation  where  the  reslduala  are  weighted  in 
a  systematic  manner.  The  oomputed  weights  describe  the  relative  importance  of 
the  data  points.  Por  example,  a  aero  weight  should  indicate  exclusion  of  a 
point.  The  removal  of  outliers  (bad  data)  will  essentially  define  robust 
data.  The  robust  procedures  applied  in  this  paper  involves  using  both  the 
N-esttmetlng  technique  of  Ruber  1*1  and  Andrews  (71.  Initially  the  Huber 
technique  is  applied  in  order  to  determine  a  robust  location  parameter 
(weighted  mean).  The  Andrew's  function  is  then  applied  using  location  para¬ 
meter  estimated  from  the  Huber  result.  Zt  should  be  noted  that  this  robust 
method  requires  a  uni-modal  distribution  of  the  data,  therefore  initial  appli¬ 
cation  of  the  Quantile  Box  Plot  should  be  made  inorder  to  establish  uni- 
modality. 

The  Huber  m-eatlaatlon  technique  whioh  involveu  defining  the  likelihood 
funtion 


L(*)  ■  -jp  fCXj-*),  (1) 

1-1 

where  f  is  a  contaminated  normal  distribution, 

Xi  -  date, 

6  *  location  parameter  and 

N  -  sample  sise 

by  maximising  log  L  ( 8  )  such  that 
whare  l>  -  f  */f 

A 

than  the  solution  of  (2)  is  ML  estlmetss  of  8  designated  as  9  .  In  ordsr  to 

rsprsssnt  i>  in  scale  invariant  form,  equation  (2)  can  ba  rewritten  ee 
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i 

£*(<!£>) .  o 

(3) 

with  d  equal  to  the  estimate  of  aoala.  The  acala  it  often  defined  aa  S 

d  -  aadlan  I  -  median  (Xi) I  /.6745 

or  alapla  M.A.D./.6745  (4) 

This  aatlaata  la  oonaldarabla  aora  robuat  than  uainq  tha  complete  aaaplaa  : 

which  oould  raault  In  poor  raapraaantation  of  tha  aotual  acala.  j 

t 

By  solving 

i  ■  °  i 

1-1  ,5, 


where 


■■-.(¥)/  m 

a  .  /*■  in  *  C, 

7  jCj  sign  (r)  in  >  cj 


I 


Cj  la  defined  aa  tha  tuning  oonatant  and 


An  itacativa  prooeaa  la  than  uaed  In  tha  aolutlon  of  (5)  auch  that  whan 
tha  dlfferanoae  in  Hx  become  negligible  therefore  providing  tha  necaaiary 
criteria  for  an  acceptable  aolutlon  for  the  8  and  Wj_  valuaa.  for  c^  ■ 

1-  345  tha  Buber 'a  <1*  funotion  provides  a  95»  efficiency. 

With  estimate  of  8  determined  from  tha  solution  of  (2)  tha  Iteration  is 
continued  where  tha  4*  function  is  now  defined  as 


nr, 


(6) 


This  new  function  is  called  the  Andrew's  wave  aquation.  In  order  to  obtain 
tha  desired  robust  data  for  this  4 ,  function,  tha  tuning  was  adjusted  to 
ox  ■  1.345  and  tha  scale  defined  aa  in  aquation  (4). 

It  should  be  noted  that  Andrew's  function  was  selected  for  its  ability  to 
describe  outliers  as  data  with  essentially  sero  weights. 
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k  general  description  of  the  Quantile  Box  Plot  ia  ahovn  in  Figure  1. 
Nhara  th«  quantile  function  is  defined  aa 


Q  (u)  •  P’4  (u),0H(l 


that  is,  if  tba  random  variabla  x  with  distribution  function  given  by  F(x>, 
tnan  tha  root  of  F(x)  •  u,  0  f  u  1  1  it  the  pin  quantila  of  r  (x) .  Prom 
tha  ordered  statiatio  X.  %  x.f  .  ,  .  ,  X ,  Q  is  dafinad  as  plece- 

visa  linaar  function  with4 ineetval  (0,1)  divided  into  2n  aubintarvala. 
Tharafora  representing  Q  as 


j  •  1.  2, 


Zn  ordar  in  interpolate 


“ ( 

Q(u)  •  ri  (u  -  ♦  n  ( -  u  ) 


whara  n  equals  tha  sample  alia. 

Tha -box  bcundarias  ara  dafinad  aa 

Q  (.25)  to  Q  (.75) 

Q  (.125)  to  Q  (.875) 

Q  (.0625)  to  Q  (.9375) 

Tha  Quantila  funotion  Q  (u)  is  uaaful  for  dataatinq  tha  praaanca  of  out- 
liars,  sodas  and  tha  existince  of  two  populations.  Plat  slots  in  Q(u) 
indioata  modas.  Sharp  risaa  in  Q(u)  for  u  naar  0  or  1  suggest  outlieraj  sharp 
risas  in  Q(u)  within  tha  boxas  indicata  tha  axistanca  of  two  (or  mora)  popu¬ 
lations.  Tha  obvious  biaodality  shown  in  figure  2  ia  raprasantad  by  tha 
Quantila  Box  Plot  displayed  in  figure  3.  in  figure  4  (lower  ordered  value) 
tha  gross  outlier  is  suggested  by  tha  extended  vertical  line  at  lower  left 
region  of  graph  in  figure  5.  The  results  shown  in  figures  (2,  3,  4,  5)  ara 
not  raprasantativa  of  typical  data  seta.  In  many  instance  multi-modality  and 
outliers  ara  not  obvious  from  routine  examination  of  tha  data. 
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U  (X  OF  RANKED  FAILURE  DATA) 

Figure  1  Quantile  Box  Plot 


Figure  3  Quantile  Box  Plot 


STRESS  DATA  EVALUATION  PROGRAM  RMS  ERROR 
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Best  Available  Copy 


Figure  4  Statistical  Data  Evaluation  of  Kelvar  Material  (Tension  Test) 


Weibull  Distribution 


Informative  Quantile  (Li )  Function 


The  objective  of  tha  IQ  function  (*1  ia  to  identify  familar  distribution 
such  as  normal  cr  Waibull  to  which  tha  statistical  rankad  data  balonga.  Thia 
mathod  providaa  an  aoourata  and  slapla  approach  to  tha  problam  of  identifying 
which  function  chould  rapraaant  tha  data.  With  tha  idantif ication  oomplatad 
than  mora  alaborata  procedures  ara  racommandad  inordar  to  provide  varif ication 
of  tha  asaumad  nodal.  If  ona  of  tha  conventional  distribution  cannot  be 
established  as  an  aocaptabla  modal  than  tha  Maximum  Penalised  Likelihood  (MPL) 
approach  is  suggested.  Tha  details  of  MPL  will  ba  discussed  later  in  tha 
text.  Applications  of  tha  ZQ  mathod  to  a  random  salaotion  of  small  sample 
sisas  (M  <  50)  from  large  sample  of  300  has  resulted  in  an  aoourata  identi¬ 
fication  of  tha  parent  population  distribution,  similar  results  ware  also 
obtained  from  larger  samples  of  1000  with  samples  of  25  and  30. 

The  Informative  Quantile  Function  is  simply  defined  as 


(10) 

where  Q(u)  was  previously  defined  in  aquation  (8).  An  example  of  IQ  vs  U  plot 
for  normal  distribution  is  shown  in  figure  (6).  Note,  at  U  ■  .02  tha  cor¬ 
responding  IQ  should  ba  approximately  -.8.  Tha  straight  line  joined  at  IQ  of 
-.5  and  .5#  raprasants  a  uniform  distribution.  This  is  introduced  in  order  to 
provide  for  an  easier  identification  of  the  unknown  distribution.  In  figure 
(7)  tha  Naibull  distribution  ia  identified/  where  0  ■  .04  and  IQ  ■  -.8. 
figure  (8)  describes  tha  form  necessary  for  identifying  tha  exponential 
distribution,  ly  generating  a  sat  of  IQ's  and  U's  from  equation  (10)  and 
plotting  these  according  to  the  figures,  idantif ication  of  tha  proper  models 
oan  be  made. 

Mamal  PUttlfriUgn  iitnaxian 
Fx  (,)  *  757  I*’  T1  dv 
where  *  •  x  "  "x 


and  x,  mx  and  ox  are  strength  data,  mean  vslua  and  standard  deviation 
respectively.  Fx  (s)  can  be  simply  and  accurately  evaluated  using  the 
following  polynomial  representation. 
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%  (*>  •  1  ♦  }  U  ♦  V  4  V2  4  dS,S  4  d4*4  4  d5*S  4  d6*6>  16  4  E  to 


where  B  (t)  < 

l.S  X  Hf7 

and  d^ 

•  .049867947 

d2 

-  .0211410061 

43 

-  .0032776263 

44 

-  .0000360036 

45 

-  .0000468806 

4« 

■  .000003383 

tegnanul 

The  lognormal  2  p»»Mt«r  distribution  function  is  svslustod  by  using 
Bquation  (11)#  short  the  aaxiaua  likslihood  ssthod  estimates  ths  assn  (ax) 
and  standard  deviation  (  ax  ) # 


v  *  i  £  h»  <*i> 

E  s,  tx4)a  -n(  £1,  exp/,)  * 


;  •  ■ 

» 


N 


*i  ■  data  valuta#  N  -  saaple  aisa 
The  unblaoad  a at lasts  of  a 


(12) 


By  defining  Lit  I,  than 

X,  ■. 

Z  •  *  ~  * 


“  {(*'*-4  O 


1/2 


*  Wtibull  Dlatrlbutlon  function 


1/2 


} 


Tha  ML  aathod  ia  applied  in  order  to  obtain  the  two  paraaatara  of  tha 
Wolbull  function 


t(x) 


($  exp  [-  ($)  ] 


(13) 
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The  method  requires  defining  the  likelihood  function!*) 


NIJi{“  (^)  ,xp['^  I 


where  Xj  ■  date, 

M,  v  ■  shape  and  mormalising  parameter  and 

N  ■  sample  else. 

By  solving  the  following  log  likelihood  equations 
»L  L 

ft—  ■  0  and 


determines  and  m  and  M  values. 

Equation  (IS)  must  be  solved  in  an  iterative  manner  by  using  the  oomputer 
oode  listed  in  Appendix  A.  The  unbiased  m  and  y  and  their  corresponding 
confidence  interval*  are  obtained  from  Tables  by  [10]. 

It  can  be  shown  thati 

®A  or°B  •*  y  (In  (A  J)  l/* 

o  (16) 

where  aK  or  °B  -  the  allowable,  depending  on  p*f, 

P*a  ■  tolerance  limit  on  P**  (probability  of  survival)  determined  from 
application  of  Monte  Carlo  method.  In  Tables  1  and  7  the  results  for  A  and  B 
allowable  and  P*g  computation  is  tabulated.  Parametric  determination  of 
three  parameter  and  aenaored  data  requires  a  more  elaborate  analysis.  These 
procedures  will  not  be  outlined  in  this  text.  The  (*  )  represents  a  biaaad 
estimate. 

Non-oar aae trie  method.  Non-parametic  procedures  t1*)  are  usually  more 
desirable  than  parametric  ones,  since  they  provide  the  exact  probabilities. 

In  the  parametric  case,  the  reliance  is  on  an  assumed  distribution  function 
which  provides  extrapolated  results  for  the  probability  of  survival  values. 
The  penalty  for  applying  the  non-pa rametrio  method  is  the  need  for  relatively 
large  amounts  of  data  (e.g.,  29  values  for  the  "B*  allowable  and  300  for  the 
"A"  allowable) .  The  lowest  ranked  value  describes  the  corresponding  allow¬ 
able.  In  the  oase  where  100  data  values  are  available,  the  sixth  lowest 
ranked  data  value  determines  the  "Ba  allowable.  The  us*  of  100  values  in 
obtaining  the  *B"  allowables  prevents  any  erroneous  estimates  if  lowest 
ordered  strength  values  are  incorrect.  Table  3  shows  which  ranked  data  value 
should  be  used  for  a  particular  sample  sise.  The  importance  of  using  sample 
sixes  greater  than  the  required  29  for  the  "B"  allowable  is  shown  in  figure 
9.  For  example,  if  all  data  is  included,  the  allowable  will  be  *4”)  however, 
this  oould  be  erroneous  if  data  value  4  was  an  outlier.  By  removing  the 
outlier,  the  allowable  t s  4.6. 


TABLE  1  Pg*  VS.  SAMPLE  SIZE  N  FOR  "A"  ALLOWABLES 


N 

V 

N 

V 

N 

V 

N 

V 

3 

. 999999 

29 

.998214 

53 

.996816 

77 

.993970 

6 

. 999992 

30 

.998130 

54 

.996776 

78 

.995940 

7 

.999972 

31 

.998048 

35 

. 996736 

79 

.995911 

8 

. 999930 

32 

. 997968 

56 

. 996697 

80 

.995883 

9 

. 999859 

33 

.997891 

37 

.996657 

81 

.995835 

10 

.999780 

34 

.997816 

58 

.996619 

82 

.995827 

11 

.999713 

35 

.997743 

59 

.996380 

83 

.995800 

12 

. 999630 

36 

.997673 

60 

. 996543 

84 

.995773 

13 

.999579 

37 

.997606 

61 

.996305 

85 

.995747 

14 

. 999300 

38 

.997541 

62 

. 996468 

86 

.995721 

15 

. 999420 

39 

.997479 

63 

. 996432 

87 

.995695 

16 

.999340 

40 

.997420 

64 

. 996396 

88 

.993670 

17 

. 999236 

41 

.997363 

65 

.996360 

89 

.993643 

18 

.999160 

42 

.997309 

66 

.996325 

90 

.995620 

19 

.999050 

43 

.997257 

67 

. 996290 

91 

.995596 

20 

. 998940 

44 

,997207 

68 

.996256 

92 

.995372 

21 

.998843 

43 

.997159 

69 

. 996222 

93 

.995548 

22 

. 998760 

46 

.997113 

70 

.996189 

94 

.995525 

23 

. 998684 

47 

.997068 

71 

.596156 

95 

.995502 

24 

.998613 

48 

.997025 

72 

.996124 

96 

.993479 

25 

. 998340 

49 

.996982 

73 

. 996052 

97 

.995456 

26 

. 998463 

50 

. 996940 

74 

« 996061 

96 

.995434 

27 

.998382 

31 

.996898 

75 

. 996030 

99 

.995412 

28 

. 998298 

52 

.996857 

76 

. 996000 

100 

.995390 
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TABLE 

2  Pg*  VS. 

SAMPLE  SIZE 

N  FOR  "B" 

ALLOWABLES 

3.0 

.99880 

29.0 

.  93991 

33.0 

.94499 

77.0 

. 93884 

6.0 

. 99783 

30.0 

.93930 

34.0 

. 94468 

78.0 

.93861 

7.0 

. 99631 

31.0 

.93861 

33.0 

.94437 

79.0 

. 93838 

8.0 

. 99430 

32.0 

.  93783 

36.0 

.94407 

80.0 

.93813 

9.0 

.99170 

33.0 

.  93703 

37.0 

.94377 

81.0 

. 93792 

10.0 

.98900 

34.0 

. 9361 7 

38.0 

. 94349 

82.0 

.93769 

n.o 

.98716 

33.0 

.93328 

39.0 

.94321 

83.0 

.93747 

12.0 

.98340 

36.0 

.93438 

60.0 

.94293 

84.0 

.93724 

13.0 

. 98287 

37.0 

.93349 

61.0 

.94267 

83.0 

.93702 

U.O 

. 97992 

38.0 

.93261 

62.0 

. 94241 

86.0 

. 93680 

13.0 

.97720 

39.0 

.93178 

63.0 

.94213 

87.0 

. 93638 

16.0 

.97317 

40.0 

.93100 

64.0 

.94190 

88.0 

. 93636 

17.0 

. 97338 

41.0 

.93029 

63.0 

.94163 

89.0 

.93614 

18.0 

. 97200 

42.0 

. 94963 

66.0 

.94140 

90.0 

.93393 

19.0 

.97013 

43.0 

.94904 

67.0 

.94116 

91.0 

.93572 

20.0 

. 96820 

44.0 

.94830 

68.0 

.94092 

92.0 

. 93332 

21.0 

. 96633 

43.0 

.94800 

69.0 

.94069 

93.0 

.93331 

22.0 

.96314 

46.0 

.94733 

70.0 

.94043 

94.0 

.93311 

23.0 

.96400 

47.0 

.94712 

71.0 

.94022 

93.0 

.  93492 

24.0 

.96306 

48.0 

.  94673 

72.0 

.93999 

96.0 

. 93472 

23.0 

. 96228 

49.0 

. 94636 

73.0 

.93976 

97.0 

. 93454 

26.0 

.96161 

30.0 

.94600 

74.0 

.93933 

98.0 

.93433 

27.0: 

.96102 

31.0 

.94366 

73.0 

.93930 

99.0 

.93417 

28.0; 

.96047 

32.0 

. 94332 

76.0 

.93907 

100.0 

. 93400 
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Figure  9  Statist ic*i  Evaluation  of  Kevlar  Material  (Hexcol) 


IWOOMTIUC  OUNHTILC  PLOT 


Multi-Modality.  If^the  bimodality  displayed  in  figures  1.  2,  and  3  la  a 
reality,  not  the  result* processing  or  testing  arrora  than  tha  currant  avail- 
abla  parametric  prooaduraa  usually  will  not  provide  aooaptabla  rapraaantation 
of  tba  data.  A  non-par see trie  aathod  (2]  i*  suggested  having  axoallant 
approximation  properties,  for  estimating  an  unknown  probability  danaity 
function  from  a  random  aampla  X^,  .  .  .  ,  XN 

Tha  aatimator  optimisms  a  oritarion  function  which  combines  tha  maximum 
likalihood  principle  and  a  penalty  term  for  smooth  (l.e.,  not  bumpy)  behavior. 
Tha  criterion  function  is  a  discrete  approximation  to 

L(f)  •  X  *(« 

*•*  L  •'I  a.1 

where  f(t)  is  any  probability  density  function!  that  is*  f(t)  is  nonnegative 
and  lntargrates  to  1.  Iquivalently,  wa  may  maximise  the  Ln(L(f)]  which 
separates  into  two  terms  -  a  log  likelihood  plus  a  log  penalty  tern. 

The  penalty  term  contains  an  unknown  positive  constant  a  which  deter¬ 
mines  the  amount  of  smoothness  in  tha  resulting  estimator.  Values  of  o  that 
are  "too  small"  result  in  bumpy  estimates  while  a  "too  large"  overanootha. 

Zn  practice,  use  an  a  as  small  as  possible  without  introducing  excessive 
bumps.  Several  values  of  <*  that  differ  by  factors  of  ten  should  be  tried  and 
graphically  displayed  and  compared  to  a  histogram  or  paranetrlo  assumption. 

Numerical  integration  determines  the  cumulative  density  values 
(probability  of  survival)  for  the  prescribe  percentile,  SO  or  9S  depending  on 
the  desired  allowable. 


*] 


(17) 


Confidence  limit  on  these  estimates  say  h.gp  or  t.go  oan  be  determine 
from  the  Cramer-Rao  lower  bound  which  determine  the  variance  on  X.  The  con¬ 
fidence  limit  L  is  determined  iteratively  from 


l 


1 


X 


1-2.  3... 


(18) 


where^Y  is  the  y  percentage  point  (951)  of  the  normal  distribution. 
Initially.  L 

I*!  -  R  -  Uy(V(R)P 


Wb#M'  V(R)  .  r2  (in  R)2  { 1.109  -  .514  la  (-1*  X)+  .608  fin  (-lnlj)2)/n 
Subsequent  lnteratlon  in  equation  IS  requires  substitution  of  L*  for  X  in 
equation  19. 


(18) 


Pooling  of  Data.  Zn  obtaining  the  allowables,  test  data  should  be  ob¬ 
tained  from  a  number  manufacturers  (e.g*  composite  materials  from  various 
airoraf t  industry  representatives) .  All  test  data  should  be  pooled  Inorder  to 
obtain  an  allowable  consistent  with  a  general  population  of  that  specific 
materials  strength  values*  Zf  a  significant  difference  exist  among  the  manu¬ 
facturer  then  an  investigation  should  be  made  regarding  the  cause  of  thla 
situation.  The  teats  recommended  for  determining  significant  differences  are 
the  conventional  t  or  Mann-Whitney  Teat#  and  two  non-parametric  tests  for  the 
K  -  sample  oase.  The  Kruakal-Wallls  multi-sample  test  for  identical 
populations  is  applied  such  that  X  is  corrected  for  ties.  The  null  hypothesis 
(identical  populations)  is  rejected  at  the  2t  level.  The  other  distribution 


frit  test,  la  tba  Jonckheere'a[12]  *  staple  trends  teat  against  the  ordered 
alternative.  Where  tiea  are  removed  by  applying  a  randomisation  process.  The 
null  hypothesis  of  randomness  is  rejected  at  the  21  level,  that  is,  aooeptance 
of  dlfferenee  in  populations  at  this  level. 

Statistical  Evaluation  of  Data  (Graphical  Display) .  In  Figure  4,  a  plot 
of  probability  failure  (Pf)  versus  failure  stress  (see  narked  circles)  of 
empirical  failure  data  were  shown.  The  Pf  values  were  determined  from  the 
Rj  ranking.  The  four  candidate  functions  are  listed  on  the  graph  with  their 
corresponding  line  fora  representations.  In  addition  to  obtaining  a  visual 
inspection  of  best  fit,  the  RMS  error  provides  a  quantitative  evaluation  for 
the  three  rankings  and  the  density  functions.  The  sample  evaluation  for  the 
three  rankings  and  the  density  functions.  The  sample  means  and  standard 
deviations  are  tabulated  with  tneir  corresponding  90%  confidence  interval. 

The  Weibull  shape  and  normalising  (Char,  value)  parameters  are  tabulated  with 
confidence  intervals.  The  991  origin  represents  one  peroent  probability  of 
failure  with  951  lower  confidence  limit  for  that  number.  Thia  representation 
is  the  A  allowable.  If  one  wants  to  lnorease  or  decrease  origin  percentage, 
it  can  be  done  by  applying  methods  described  in  (2) .  The  word  "origin"  will 
equal  sero  if  two  parameter  Weibull  funotion  was  considered,  otherwise,  three 
parameter  Weibull  function  was  used.  Zn  the  radical  parameter  tabulation,  A, 

B  and  C  are  ooeffioients  obtained  from  a  least  squared  fit  routine.  Bxp  8  (N) 
and  Bxp  C  (R)  are  the  corresponding  exponents  determined  in  the  fitting  of  the 
date.  Big  X  and  dig  F  are  the  two  out-off  point*.  That  is,  the  smallest  and 
largest  projected  values  determined  by  the  function. 

Zn  applying  the  robust  procedures,  it  is  important  to  have  a  rationale  for 
ignoring  the  determined  outlier,  otherwise,  erroneous  estimates  of  survival 
probability  computations  could  result.  The  robust  scheme  can  be  applied  to 
data  from  relatively  small  slse  specimens,  where  errors  in  machining,  twsting, 
etc. ,  greatly  effect  strength  determination.  The  authors  have  noted  consider¬ 
able  Improvement  in  ceramic  material  failure  predictions  of  large  speolmens 
(13  in3)  from  knowledge  of  small  specimen  (.03125  in3)  strength  results 
when  the  outliers  are  removed  from  the  original  data.  Non-parametrlos  solu¬ 
tions,  are  applied  to  the  code  (see  Figure  4),  aan  provide  information  regard¬ 
ing  oonfidanoe  levels  for  A  and  B  allowables,  with  respect  to  the  number  of 
data  points.  The  code  oan  be  altered  to  include  other  ellowablee  by  using  the 
simple  relationships  outlined  in  (11) .  Zn  the  last  three  boxes  the  allowable 
estimates  are  tabulated  for  the  weibull,  normal  and  lognormal  functions. 

Results  and  Discussion 

Zn  figure  4,  the  results  of  tension  tests  on  Kevlar  composite  material 
(Hexcel  Co.)  are  shown,  they  are  similar  for  all  three  functions.  This  is  an 
ideal  situation,  since  selecting  ellovables  from  any  of  the  functions  will 
provide  acceptable  results.  Zn  figure  9,  the  results  from  another  manu¬ 
facturer  using  Hexcel  material  are  shown.  The  RMs  error  indicates  the  weibull 
funtion  would  be  an  acceptable  representation  of  data.  Zn  figure  9a,  a  ZQ 
plot  of  data  from  figure  9  test  results  also  indicates  data  should  be  repre¬ 
sented  by  a  Weibull  function.  The  results  shown  in  figure  10,  describe  the 
existence  of  an  outlier  at  highest  ordered  value.  See  figure  11  display  of 
Box  Plot  for  verification  of  outlier  value  (26.5).  It  is  obvious  from  figure 
10  that  Weibull  funotion  does  not  represent  lower  tail  region  particularly 
well.  The  resultant  design  "B"  allowable  of  10.9  determined  from  Weibull 
computation  differs  from  non-parameter  solution  by  2.1  a  19%  difference.  The 
"A"  allowable  result,  Weibull  vs.  normal  is  different  by  2.1.  Sinoe  the 
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non-par ama trie  result  la  an  accurate  measure  than  tha  Weibull  function  haa 
produced  an  error  of  approximately  19%  for  at  laaat  tha  *B”  allowable  and 
poaaibla  nor#  for  tha  aA"  allowable.  Zn  figure  12#  tha  outlier  haa  bean 
removed  from  tha  data  diaplayad  in  figure  10.  Note  tha  aubatantial  inoraaaa 
in  tha  Waibull  ahapa  par ana tar  from  5.66  to  7.22  by  removing  one  outlier 
(hlgheat  ordered  value).  These  raaulta  indicate  need  for  exploring  data  prior 
to  applying  functional  repreaentation.  If  tha  outlier  actually  exiata,  than 
allowable  obtained  from  noraal  distribution  should  be  considered.  Deter¬ 
mination  of  acceptable  Waibull  parameters  from  the  ML  method  depends  on  the 
absence  of  outliers  at  highest  ordered  values.  Since  the  vulnerability  of  the 
ML  method  has  bean  exposed  in  tha  above  example,  it  suggested  that  the  Beat 
Linear  Unbiased  Estimate  procedure  also  be  used  in  order  to  determine  the 
Waibull  parameters#  thereby  providing  flexibility  in  the  selection  of  para¬ 
meter  estimating  procedures. 

Figure  13  describes  the  statistical  results  from  compression  tests  on 
Kevlar  material  obtained  from  the  three  manufacturers.  Zn  figure  14#  tha 
results  from  pooling  original  three  manufacturers  data#  with  a  forth  manu¬ 
facturer  (submitted  data  at  a  later  date).  An  approximate  61  reduction  in 
allowable  estimate  with  addition  of  the  forth  data  set.  The  are  two  fund¬ 
amental  issues  involved  one  is  the  need  for  a  substantial  number  of  manu¬ 
facturers  partlolpsting  in  the  allowables  computation  program  and  seoondly  the 
possible  reliance  on  an  extreme  value  type  (Waibull)  distribution  in  order  to 
introduce  conservatism  when  accounting  for  the  uncertainties  existing  from  a 
limited  number  of  pooled  samples  in  representing  the  population. 

The  Kruakal-Wallis  test  for  determining  K  -  sample  difference  indicated 
the  forth  monufaoturer  data  differed  slgnif loantly  from  tha  other  samples  at 
.01  levels.  At  present,  the  MXL-HDBK-17  recommends  pooling  all  samples  unless 
a  rationale  has  been  established  for  removal  of  sample. 

Bven  though#  homogeneous  data  is  not  available  for  determining  allowables# 
the  committee  considers  it  more  important  to  represent  the  difference  among 
manufacturers#  particularily  so  for  composite  materials. 

In  figure  15#  the  higher  ordered  values  appeared  to  be  from  a  different 
mode  of  failure.  The  Box  Plot  example  shown  in  figure  1  is  a  display  of  this 
data  which  essentially  oonfirms  the  existence  of  blmodallty.  It  should  be 
noted  in  figure  15  that  none  of  the  candidate  functions  adequately  represent 
the  ranked  data.  The  allowables  are  too  conservative#  particularly  for  the 
Weibull  and  normal  representative.  A  suggested  alternative  in  determining 
allowables  for  this  data  is  the  Maximum  Penalised  Likelihood  Methodic 
described  in  the  text.  The  results  are  shown  in  this  figure  are  an  excellent 
repreaentation  of  the  ranked  data  including  the  sizable  bump.  The  allowables 
of  5.04  ksi  for  the  *Aa  and  5.51  ksi  for  the  "B".  The  "B*  allowable  agrees 
within  .51  of  the  non-parametric  solution.  At  present#  the  authors  consider 
this  the  most  acceptable  method  for  determining  allowables  where  multi- 
modality  exists  in  the  data,  examination  of  other  methods#  such  as  consid¬ 
ering  upper  mode  as  censored  data  or  application  of  the  mixed  Weibull  distri¬ 
butions  to  the  data#  proved  to  be  inadequate.  The  later  method  may  have  merit 
if  selected  percentiles  of  the  distribution  are  matched  with  the  corresponding 
ranked  values  in  a  manner  that  guaranteed  a  good  fit.  in  figure  16  another 
example  of  bimodality  is  shown.  Zn  this  case#  representation  of  Weibull  shape 
parameter  value  of  6.75  is  extremely  low.  The  results  from  MPL  method  are 
also  shown  in  the  figure  including  tabulation  of  the  allowables.  Manufacturer 
of  this  material  has  recently  indicated  that  lower  mode  data  was  incorrectly 
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added  to  iqppar  node  data  dua  to  different  autoolavaa  uaad  In  processing  of 
material. 

Figure  17  shows  tha  raaulta  fron  Tanaion  taat  on  Ravlar  material  (Cycoo 
Co.).  Tha  allowable  computations  differ  by  at  moat  7%  for  tha  three 
functional  repraiantationa.  Allowable!  determined  from  normal  computation 
would  be  animated  for  thia  material  and  taat. 

Figure  18.  ahowa  data  evaluation  of  Composite  graphite  material.  The 
existence  of  bimodality  displayed  in  the  sample  was  a  common  occurence  among 
several  of  the  graphite  test  samples. 
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STRESS  DATA  EVALUATION  PROGRAM  RMS  ERROR 


Figure  13  Statistical  Data  Evaluation  Kevlar  (Compression  Test)  Partial  Sample 
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STRESS  ‘  ICS  I ) 

Figure  17  Statistical  Data  Evaluation  Composite  (CYCOM)  Tension 


Conclusion 


1.  Exploratory  Data  Procedure  should  ba  applied  prior  to  acceptance  of 
statistical  model  used  in  the  allowable  computation. 

2.  Quantile  Box  Plot  provides  an  excellent  summary  of  test  data  results 
in  addition  to  location  outliers  and  multi-modality  in  the  sample. 

3.  The  authors  rocotmand  using  the  Welbull  distribution  function  for 
obtaining  the  allowables,  if  outliers  (higher  ordered  values)  or 
multi-modality ,  do  not  exist  in  data  set. 

а.  An  extreme  value  distribution  (Welbull)  provides  a  degree  of  security 
if  pooled  samples  do  not  represent  general  data  population,  (a 
common  occurrence) . 

3.  Nortal  distribution  is  recommended  for  allowable  computation  if 
outliers  exist  in  data  set  at  upper  tail  region. 

б.  In  multi-modality  case,  the  Maximum  Penalized  Likelihood  Method 
le  suggested  for  the  allowable  determination. 

7.  At  present  the  authors  recommend  pooling  all  samples  made  available 
even  though  significant  difference  teat  indicated  otherwise.  If 

a  sample  contains  modality  then  this  data  set  would  be  excluded. 

8.  The  development  uf  tables  1  and  2  required  a  considerable  amount 

of  effort  (most  complete  tabulation  for  tha  Welbull  function) .  With 
the  aid  of  computer  code  listed  in  appendix  A  and  tabulated  P*  values, 
the  reader  should  be  able  to  obtain  a  simple  and  accurate  commutation 
of  the  allowables  when  the  Welbull  function  is  considered. 

9.  Non-parametric  procedures  are  always  the  most  desirable  in  obtaining  the 
allowables  for  the  given  sample  if  properly  applied. 

10.  A  sufficiently  large  number  of  sources  in  obtaining  test  data  is  more 
important  in  determining  allowables  than  size  of  individual  sample. 
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WEXBULL  PARAMETERS  M  AND  U 


READ8  XN  AMOUNT  OF  DATA  AND  THE  DATA 
DIMENSION  T ( 200) 

IMPLICIT  DOUBLE  PRECISION  <A-H»0-Z> 
READOf  1  )NPT8  < — 
READ<3»1XT(I)»  !■!  >NPT8>  < - 


•Sampla  Siz* 


•Data 


DETERMINES  M  BY  MAXIMUM  LIKELIHOOD 
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N-0.0 

DA-1,0 

XNUM-0,0 

XNXNU8-0.0 

PRQDl-0,0 

PR0D2-0.0 

A-A+DA 

DONATIONS  CALCULATED  FIRST 

DO  10  1-1 »NPTS 

XNUM-T  < I ) tXA+XNUM 

XMINU8-T  < I ) **A*L0G <  T ( I ) ) +XMINU8 

PR0D2-L00 ( T (I ) ) +PR0D2 

A1-NPTS4XNUM/ <NPTS*XMXNUS-XNUM*PR0D2 ) 

IF< A-Al )2»3»3 

N-Ntl 

IF(N-5)4» 4»3 
A-A-DA 
DA-1, /10,**N 
A-A-DA 

00  TO  2  * 

«RITE(4*1)A  < - m 
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DETERMINES  U  'NAT' 


B8UM-0.0 
DO  4  I-lrNPT8 
B8UM-TU)«*A+&SUM 
B-(1./NFT8*BSUM)**<1 ,/A) 

WRITEf 1 )B  < - 

FORMAT  <  ) 

END 


Appendix  A  Computer  Code  for  Determining  Weibull  Parameters 


235 


CHOICE  OP  RESPONSE  SURFACE  DESIGN 
AND  ALPHABETIC  OPTIMALITY 

George  E.  P.  Box 

UNIVERSITY  OP  WISCONSIN  -  MADISON 
MATHEMATICS  RESEARCH  CENTER 

ABSTRACT 

It  is  erqued  that  the  specification  of  problems  of  experimental  design 
(and  in  particular,  of  response  surface  design)  should  depend  on  scientific 
context.  The  specification  for  a  widely  developed  theory  of  "alphabetic 
optimality"  for  response  surface  applications  is  analysed  and  found  to  be 
unduly  limiting.  Ways  in  which  designs  might  be  chosen  to  satisfy  a  set  of 
criteria  of  greater  scientific  relevance  are  suggested.  Detailed  considera¬ 
tion  is  given  to  regions  of  operability  and  interest,  to  the  design  information 
function,  to  sensitivity  of  criteria  to  siso  and  shape  of  the  region,  and  to 
the  of foot  of  bias.  Problems  are  discussed  of  checking  for  lack  of  fit, 
sequential  assembly,  orthogonal  blocking,  estimation  of  error,  estimation  of 
transformations,  robustness  to  bad  values,  using  minimum  numbers  of  points, 
and  employing  simple  data  patterns. 
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Choice  of  Response  Surface  Design  and  Alphabetic  Optimality 


George  E»  P.  Box 


University  of  Wisconsin  -  Madison 


1 .  INTRODUCTION 

Thsrs  sssns  no  doubt  that  of  all  tho  activities  in  which  the 
statistician  can  engage,  that  of  designing  experiments  is  by  far  the  most 
important#  since  it  is  here  that  ths  aotual  node  of  generation  of  scientific 
data  is  decided* 

The  importance  of  practice  in  guiding  the  developeent  of  the  theory  of 
experimental  design  (4S]  is  clearly  seen  from  the  time  of  its  invention. 
Pisher  was  engaged  by  Russell  (16]  on  a  temporary  basis  at  Rothamstad 
Experimental  Station  in  1919  "to  examine  our  data  and  elicit  further 
information  that  we  had  missed."  Records  were  available  from  the  ongoing 
BroadbalX  experiment  in  which  particular  combinations  of  fertilisers  had  been 
consistently  applied  to  13  plots  for  a  period  of  almost  70  years.  In  his 
analysis  ((221,  [24] ),  Pisher  attempted  to  relate  yield  to  fertiliser 
combination,  to  weather,  and  in  particular,  to  rainfall.  The  method  he  used 
was  multiple  regression  with  distributed  lag  models,  involving  an  ingenious 
employment  of  orthogonal  polynomials  which  led  to  important  advances  in  the 
theory  of  regression  analysis,  and  in  particular  its  distribution  theory. 


Spor.sorod  by  the  United  States  Army  under  Contract  No.  DAAG29-80-C-0041 . 


With  only  the  crudest  of  computations!  aids*  the  work  must  have  been 
burdensome,  making  it  all  the  more  frustrating  to  discover  that,  however 
ingenious  the  analysis,  the  inherent  nature  of  the  data  ensured  that  the 
answers  to  many  questions  were  inaccessible.  A  comprehension  of  the  logical 
problems  in  drawing  conclusions  from  such  analyses  led  naturally  to 
speculation  on  how  some  of  the  difficulties  might  be  overcome  by  appropriate 
ilirii'in.  idoiiu  worn  I'urUiur  :it.  imulutwi  by  Uiu  Analysis  of  Variance, 

which  Flutter  Introduced  in  1923  with  W*A*  Mackenzie  (23)  for  the  elucidation 
of  what  was  clearly  a  most  unsatisfactory  design  whioh  he  had  had  no  part  in 
choosing.  Thereafter,  as  Fisher  gradually  acquired  more  influenoe  in  the 
setting  up  of  field  trials,  the  principles  of  replication,  randomisation  and 
their  application  to  randomised  blocks,  latin  squares  and  factorial  designs 
quickly  evolved  out  of  the  actual  planning,  running,  and  analysis  of  a  series 
of  experimental  designs  of  increasing  complexity  and  baauty. 

Tha  praotloal  context  of  scientific  experimentation  continued  to 
produce  important  theoretical  advances  whtn  Yatss  cams  to  Rothamstad  in  1931, 
leading  in  particular  to  important  developments  in  tha  design  and  analysis  of 
aomplex  factorial  designs  and  thsir  associated  systems  of  confounding  ((44), 
(46))  and  to  the  introduction  of  incomplete  block  designs. 

My  own  experience  with  experimental  design  began  during  tha  Saoond 
world  War.  I  worked  at  the  Chemical  Defense  Experimental  Station  in  England 
with  a  group  of  medical  research  workers  who  were  attempting,  using  animals 
and  volunteers,  to  find  ways  to  combat  the  effects  of  poison  gss  end  other 
toxic  agents.  At  this  time  it  was  believed  that  these  agents  might  be  used 
not  only  against  the  military,  but  also  against  the  civilian  population.  It 
was  important  therefore  that  our  work  should  progress  as  rapidly  as 
possible.  I  found  myself  a  part  of  evolving  investigations  which  employod 


sequences  of  experimsnts  which  1  designed  and  whose  nature  needed  to  adapt  to 
changing  needs  at  different  stages  of  the  study.  The  designs  employed  were 
randomised  blocks,  balanced  incomplete  blocks,  latin  squares,  and 
factorials*  Later,  during  my  eight  years  as  a  statistician  with  Imperial 
Chemical  Industries,  my  role  was  again  as  a  member  of  various  scientific  teams 
tackling  evolving  problems',  wi th  sequences  of  dssigns*  Many  of  ths  problems 
were  similar  to  those  1  had  previously  encountered,  end  again  employed  the  (by 
now)  standard  designs  of  Fisher  and  Yates.  However,  some  investigations 
directly  concerned  with  the  improvement  of  ohemioal  processes  at  the  lab, 
pilot  plant,  and  full  scale,  seemed  to  r^uire  additional  methods,  which 
however,  still  drew  on  the  fundamental  principles  laid  down  by  ths  originators 
of  experimental  dasign.  This  lad  to  the  development  of  whet  has  come  to  be 
oalled  response  surface  methodology.  See  for  example  (4),  (14),  (15),  (30), 
(31),  and  (39). 

Suppose  some  response  n  of  interest  is  believed  to  be  looelly 
approximated  by  a  polynomial  of  low  degraa  in  k  continuous  axperlmental 
variables  X  ■  (X,,  Xa»  •  ••,  X^) * •  To  fit  euoh  a  function  we  need  appropriate 
experimental  designs.  Let  us  call  a  design  suitable  for  estimating  a  general 

polynomial  of  degree  d  a  dth  order  design  in  k  variables.  Thus  a  design. 

\ 

suitable  for  fitting  the  funotion 

n  "  *0  *■  ®1X1  *  ®2X2  *  811X1  +  *22X2  +  *12X1X2 

would  be  a  saoond  order  design  in  two  variables. 

Ons  route  for  choosing  such  designs,  which  has  generated  an  enormous 
amount  of  mathematical  research  over  the  last  twenty  or  so  years,  we  shall 
refer  to  as  the  "alphabetic  optimality”  approach.  For  reasons  I  will  explain, 
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I  have  reservations  about  the  usefulness  of  this  approach  so  far  as  response 
surface  designs  are  concerned.  For  completeness,  a  brief  summary  of  some  of 
the  main  ideas  are  set  out  below  ((32),  (33],  (34],  (35],  (36),  (42],  (43)). 

2.  SOME  ASPECTS  OF  OPTIMAL  DESIGN  THEORY 
FOR  CONTINUOUS  EXPERIMENTAL  VARIABLES 

Consider  a  response  h  which  is  supposod  to  be  an  exactly  known 
function  n  ■  x'0  linear  in  p  coefficients  0,  where 

«*  •  m 

*  ■  ^(X)#  f^U)#  •••#  f  <x»*  i*  *  vector  of  p  functions  of  k  experimental 
variables  x«  Suppose  s  design  is  to  be  run  defining  n  sets  of  k  experimental 
conditions  given  by  the  n  x  k  design  matrix  {x)  end  yielding  n  observations 

eU 

(yu),  so  that 


\  ■  *'•  (u  *  1,  2,  ...,  n) 

VI  «U« 

where  y  -  n  ■  c  is  distributed  N(0,o2)  and  the  n  x  p  matrix  x  -  {x'}. 

M  *  m  «U 

The  elements  of  {o^}  -  (X'X)"1  are  proportional  to  the  variances  and 

A 

covariances  of  the  least  squares  estimates  0.  Within  this  specification,  the 

a* 

problem  of  experimental  design  is  that  of  choosing  the  design  (x  }  so  that 

eU 

the  elements  are  to  our  liking.  Because  there  are  V{fp(p+1 )  of  these, 
simpllf ioatior.  is  desirable. 

A  motivation  for  simplification  is  provided  by  considering  the 
confidence  region1  for  0 


Obviously  there  are  also  parallel  fiducial  and  Bayesian  rationalisations. 
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(0  -  0)'X'X(0  -  0)  ■  constant 

*»  rn  m  rnt  m  m 

defining  an  ellipsoid  In  p  parameters.  The  eigenvalues  X^ ,  X^,  . ...  X^  of 
(X'X)-1  are  proportional  to  the  squared  lengths  of  the  p  principal  axeo  of 
this  ellipsoid.  Suppose  their  maximum,  arithmetic  mean,  and  geometric  mean 
are  indicated  by  ^MX'  and  X.  Then  it  is  illuminating  to  consider  tha 
transformation  of  the  yp(p+1)  elements  c^  to  a  corresponding  number  of 
items  as  followst 

(i)  D  •  | X* X |  •  X’p  (so  that  ■  Xp^*  is  proportional  to 

the  volume  of  the  confidence  ellipsoid). 

(ii)  H.  a  vector  of  p  -  1  homogeneous  funotions  of  degree  sero  in 

•e 

the  X'e.  which  measure  the  non-sphericity  or  state  of  ill- 
conditioning  of  the  ellipsoid.  In  particular  we  might  choose,  for 

m  m 

two  of  these.  H  ■  X/X  and  -  X  /X  .  both  of  which  would 
take  the  value  unity  for  a  spherical  region. 

(ill)  V*fp(p-D  independent  direction  oosines  which  determine  the 
orientation  of  the  orthogonal  axes  of  the  ellipsoid. 

It  is  traditionally  assumed  that  the  p(p-1 )  elements  ooncerned  with 
orientation  of  the  ellipsoid  ars  of  no  intersst,  and  attsntlon  has  besn 
conosntratsd  on  partioular  oritsria  which  measure  in  some  way  or  another  the 
sisee  of  the  eigenvalues,  measuring  some  combination  of  sise  and  sphericity  of 
the  confidence  ellipsoid.  Among  these  oriterie  ere 

o  -  j x 1 x l  -  nx"1  -  x"p 

A  »  IX  •  tr(X'X)  1  -  0~*Evar(0.  )  ■  pXH, 

*  e  e  JL  1 

X  «  max(A^)  ■  XH^ 
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The  desirability  of  a  design,  as  measured  by  the  0,  A,  and  E  criteria, 


increases  as  X,  Ah^,  and  Ah2  respectively,  are  decreased.  But  in  practical 
situations,  each  of  these  criteria  will  take  smaller  and  hence  more  desirable 
values  as  the  ranges  for  the  experimental  variables  x  are  taken  larger  and 
larger.  To  oope  with  this  problem  it  is  usually  assumed  that  the  experimental 
variables  x  may  vary  only  within  some  exaotly  known  region  in  the  spaoe  of 
X,  but  not  outside  it.  I  will  call  this  permissible  region  RO. 

<■  a*  «• 

Another  characteristic  of  the  problem  which  makes  its  study 
mathematically  difficult  is  the  necessary  discreteness  of  the  number  of  runs 
which  can  be  made  at  any  given  location.  In  a  technically  brilliant  paper 
(37],  Kiefer  and  Wolfowits  dealt  with  this  obstaole  by  introducing  a 
continuous  design  measure  {  whioh  determines  the  proportion  of  runs  which 
should  ideally  be  made  et  etch  of  e  number  of  points  in  the  x  space* 

m 

Realisable  designs  whioh  most  nearly  approximated  the  optimal  distribution 
could  then  be  used  in  practice. 

A  further  important  result  of  Kiefer  and  Wolfowits  linked  the  problem 
of  estimating  B  with  that  of  estimating  the  raeponse  n  via  the  property  of 

m 

"O-optimality."  G-optimal  designs  were  defined  as  those  which  minimised  the 

A 

maximum  value  of  V(y  .)  within  RO.  The  authors  were  than  able  to  show,  for 

X  ***",““  «e  m 

their  measure  deeigne,  the  equivalence  of  G-  and  D-optimality.  Furthermore, 
they  showed  that,  for  euoh  e  design,  within  the  region  RO,  the  maximum  value 

m  «e 

*  2 

of  n*Var(yx)/o  was  p,  end  that  this  value  wee  actually  attained  at  each  of 
the  design  points. 

For  illustration  we  consider  «  second  order  measure-design  in  two 

variables?  that  is,  a  design  appropriate  for  the  fitting  of  the  second  degree 

polynomial  of  equation  (1).  Such  a  design  which  Is  both  0-  and  G-  optimal  for 

a  square  region  RO  with  vertices  (±1,11)  was  given  by  Fedorov  (21]  (see 
•* 


'jwmsx- 
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•Iso  He rt berg  (27)).  The  design  places  14,6%  of  the  measure  at  each  of  the 
four  vertices#  8,0%  at  each  of  the  midpoints  of  the  edges,  ar.d  9.6%  at  the 
origin.  The  design  is  set  out  in  Figure  4(b). 

While  this  approach  hau  generated  much  interesting  mathematics,  it  do«9 
not,  I  believe,  solve  the  problem  of  choosing  good  response  surface  designs. 

Zn  the  hope  of  stimulating  paw  initiative,  I  have  set  out  below  what  I  believe 
is  the  scientific  context  for  response  surface  studies  and  indicated  soma 
possible  lines  of  development. 

3.  TUB  RESPONSE  SURFACE  CONTEXT' 

As  sn  example  suppose  it  is  desired  to  study  some  chemical  systsm,  with 
the  object  of  obtaining  s  higher  value  for  a  response  n  suoh  as  yield  which 
is  initially  believed  to  be  some  function  n  -  g(x)  of  k  continuous  input 
variables  x  «  (Xj,X2»»»*,X|t)'  euch  as  reaction  time,  temperature,  or 
concentration.  As  is  illustrated  in  Figure  1,  it  la  usually  known  initially 
that  tha  system  can  be  operated  at  soma  point  xn  in  the  space  of  x  and  is 
expected  to  be  capable  of  operating  over  some  much  more  ox tensive  region 
0  called  the  operability  region,  which2  however  is  usually  unknown  or  poorly 
known*  Response  curface  methods  are  employed  when  the  nature  of  the  true 
response  function  n  -  g(x)  is  also  unknown3  or  is  inaccessible. 

m 


2 

One  secondary  object  of  the  investigation  may  bs  to  find  out  more  about  the 
operability  region  0. 

“Occasionally  tha  trua  functional  form  n  -  g(jp  may  be  known,  or  at  least 
conjectured,  from  knowledge  of  physical  mechanisms.  Typically  however  g(]/) 
will  then  eppear  as  a  solution  of  a  set  of  differential  equations  which  are 
nonlinear  in  a  number  of  parameters  whioh  may  represent  physical  constants. 
Problems  of  nonlinear  experimental  design  then  arise  which  are  of  considerable 
interest  although  they  have  received  comparatively  ILttle  attention  (hou  Cor 
example  (13),  (18),  (26)). 
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Figure  1.  Th«  current  region  oif  interest  R  and  the 
region  of  operability  0  in  the  space  of 
two  continuous  experlaental  variables  X, 
and  Xa. 
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Suppose  that  over  some  (typically  much  less  extensive)  immediate  reg ion 
of.  interest  R  in  the  neighborhood  of  xQ  it  i«  guessed  that  a  "graduating" 
function,  such  as  a  dth  degree  polynomial  in  x, 

\  *  x'0 

A  mm, 

might  provide  a  locally  adequate  approximation  to  the1  true  function 

1  -  g(X)  where  as  before  x  is  a  p-ditnensional  vector  of  suitably 

X  -  * 

M 

transformad  input  variables  x*  ■  (f  (x),f_(x)» « * • »f  (X)}/  and  ()  is  a  vector 

m  1  e  2  «  P  «*  - 

of  coefficients  occurring  linearly  that  may  be  adjusted  to  approximate  tho 

unknown  true  response  function  ■  g(x)»  Then  progress  may  be  achieved  by 

using  a  sequence  of  such  approximations.  For  example  when  e  first  degree 

polynomial  approximation  could  be  employed  it  might,  via  the  method  of 

stewpeat  ascent,  be  ueed  to  find  a  new  region  of  interact  where,  aay,  the 

yield  was  higher.  Alao  a  maximum  in  many  variablea  le  often  represented  by 

4 

some  rather  complicated  ridge  system  and  e  second  degree  polynomial 
approximation  when  suitably  analysed  might  be  used  to  elucidate,  describe,  and 
exploit  such  a  eystem. 

Thus  we  are  typically  involved  in  using  e  sequence  of  designs,  each 
making  use  of  information  gleaned  from  earlier  experiments  —  a  characteristic 
typioal  of  a  much  wider  field  of  scientific  investigation.  This  provides  the 


4 

Empirical  evidence  suggests  this.  Also,  integration  of  sets  of  differential 
equations  which  describe  the  kinetics  of  chemical  systems  almost  invariably 
leads  to  ridge  systems  ((4),  (IS),  (2b],  (41)).  See  also  the  discussion  of 
l-’ujure  8. 
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opportunity  to  progressively  improve  not  only  the  objective  function  n 
directly,  but  also  the  mode  of  gathering  information  about  it.  For  example, 
at  the  _ith  stage,  a  design  performed  in  a  region  may  suggest  that  a  new 
region  R^+1  is  worthy  of  investigation  (either  because  it  can  be  expected  to 
give  higher  values  of  n  or  because  it  may  throw  light  on  other  important 
aspects  of  the  function).  But  this  new  region  may  be  different  not  only  in 
(a)  its  location  in  the  space  of  X*  but  (b)  in  its  shape  also  (for  instance 
because  of  information  fed  baoh  from  previoue  data  on  transformations  of 
X's  individually  or  jointly),  and  (c)  in  the  identity  of  its  component  space 
(because  of  feedback  from  the  results  themselves,  indicating  that  certain 
variables  should  be  dropped,  and/or  that  new  variables  should  be  added).  Thus 
in  any  realistio  view  of  the  process  of  investigation  the  dimensions, 
identity,  location  and  metrics  of  measurement  of  reqions  of  interest  in  the 
experimental  space  ere  all  iteratively  evolving.  The  problem  of  choosing 
suitable  experimental  designs  in  such  e  context  ia  a  difficult  one.  Soma 
properties  ((5),  [8])  of  a  response  surface  design,  any,  all  or  some  of  which 
might  in  different  circumstances  be  of  importance  in  the  above  context  are 
given  in  Table  1 . 

Tha  design  information  function 

Associated  with  requirements  (1)  and  (2)  of  Table  1,  consider  the 
design  variance  function  [11) 

v  -  n»Var (y  )/o2»  nx'U'xT’x 

X  X  m  rn  m  m 

«  m 

or  equivalently  the  Information  Function 
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The  design  should: 

(i)  genoreto  »  satisfactory  distribatioa  of  information  throughout  tho  region  of 
interest,  R\ 

(ii)  ensure  thet  the  fitted  value  et  *,  0(»)  he  u  cloee  ec  poeeible  to  tho  true  Ttlue  et  *, 
?(*); 

(Hi)  gi to  good  detectability  of  look  of  fit; 

(It)  allow  transformation#  to  be  estimated ; 

(?)  allow  experiments  to  be  performed  in  blocks; 

(▼i)  allow  designs  of  increasing  ordor  to  be  built  up  sequentially; 

(vil)  provide  an  internal  estimate  of  orror; 

(viii)  bo  insensitive  to  wild  observations  and  to  violation  of  the  usual  normal  theory 
assumptions; 

(be)  require  a  minimum  number  of  experimental  points; 

(x)  provido  simple  data  patterns  that  allow  ready  visual  appreciation ; 

(xl)  ensure  simplicity  of  oaleulation ; 

(xii)  behave  well  when  errors  occur  in  the  aottinga  of  the  predictor  variables,  tho  r't; 
(xiii)  not  require  en  impreotioaliy  large  number  of  prediolor  varlablo  levolt; 

(xiv)  provido  a  cheek  on  the '  const  snoy  of  variance  ’  eeeumption. 


TABLE  1. 


SOME  ATTRIBUTES  OF  DESIGNS  OF  POTENTIAL  IMPORTANCE 


It  is  evident  that  if  we  wara  to  make  tha  unraalistic  assumption  (mado  in 
alphabetic  optimality)  that  tha  graduating  function  n  ■  x'0  is  capable  of 

a  •» 

exactly  rapraaontlng  tha  true  function  g(x)  ,  than  tha  information  function 
would  tall  us  all  we  could  know  about  the  design's  ability  to  estimate  n. 

For  illustration,  information  functions  and  associated  information  contours 

2  2 
for  a  2  factorial  used  as  'a  first  order  design  and  for  a  3  factorial  used  as 

a  second  order  design  aro  shown  in  Figures  2  and  3,  for  standard  variables  X, 

and  )<2* 


4.  APPLICABILITY  OP  ALPHABETIC  OPTIMALITY 


The  Information  function  for  Fedorov's  seoond  order  D/G-optimal  design 
over  the  permissible  RO  region  (±1,*1),  referred  to  earlier,  is  shown  in 

•  m 

Pigura  4.  For  illustration,  this  is  related  to  tha  two  experimental  variables 
X1  -  ts^p  in  °C  and  x2  ■  tims  in  hours.  Thus,  in  this  particular  example, 

*1  •  (Xj  -  180)/10,  X2  “  X2  *  *  *nd  R0  r*^ion  woul<1  permit 

experimentation  within  the  limits  X1  ■  170  -  190  *C  and  x2  ■  3  -  5  hours, 
but  not  outside  these  limits.  In  the  response  surface  context  a  number  of 
questions  arise  concerning  the  appropriateness  of  the  specification  set  out  in 
Section  2  of  this  paper  for  alphabetic  optimality.  These  concern 

(i)  Formulation  in  terms  of  the  RO  region 

(ii)  Distribution  of  information  over  a  wider  region 

(iii)  Sensitivity  of  criteria  to  flizo  And  stiApt  of  th6  RO  ro^ion 

*»  ** 

(iv)  Ignoring  of  bias. 
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?igur«  2(»)  Information  surfaca  for  a  22  factorial  used 
as  a  first  ordsr  dssign. 
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Figurs  2 (b)  Information  contours  for 
as  a  first  order  design. 


"2 •  —  1 «  0 •  1  i 


Pigura  3(b)  Information  oontours  for  *  3 2  factorial 
a  aacond  ordar  daaign. 
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Formulation  in  terms  of  the  RQ  region 


As  has  bean  pointed  out,  in  response  surface  studies  it  is  typically 
true  that  at  any  given  stage  of  an  investigation  the  current  region  of 
interest  R  is  much  smaller  than  the  region  of  operability  0  which  is,  in  any 

m  ■» 

case,  usually  unknown.  In  particular,  it  is  obvious  that  this  must  be  so  for 
any  investigation  in  which  we  allow  the  possibility  that  results  of  one  design 
may  allow  progress  to  a  different  unexplored  region.  Consequently  X  believe 
that  formulation  in  terms  of  an  RO  region  which  assumes  that  R  and  0  are 

mm  mm 

identical  is  artificial  and  limiting.  Xn  particular,  to  obtain  a  good 

approximation  within  R  one  may  very  well  wish  bo  put  some  experimental 
■  .  1 

points  outoid*  R  and  so  long  aa  they  ara  within  0  thara  la  no  praotloal 

—  Ml 

reason  why  we  should  not.  Also  einoe  typically  R  is  only  vaguely  known,  we 
will  want  to  consider  the  information  function  over  a  wider  region,  as  is  done 
for  example  in  Figure  5  for  Pedorov's  second  order  D-optimal  design.  The 
information  function  for  this  design  may  now  be  compared  over  this  wider 
region  with  that  for  the  3 2  factorial  in  Figure  3. 

Distribution  of  information  over  a  wider  region 

In  the  response  surface  context,  the  coefficients  P  of  a  graduating 

m 

function  n  -  x'0  acting  as  they  do  merely  as  adjustments  to  a  kind  of 
X  *  * 

mathematical  french  curve  are  not  usually  of  individual  interest  except 
insofar  as  they  affect  n,  in  which  oase  only  the  C-optimality  criterion  among 
those  considered  is  of  direct  interest.  For  response  surface  studies  however, 
it  is  far  from  clear  how  desirable  is  ths  property  of  G-optimality  itself. 
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For  instance,  the  profiled  of  Figure  6  made  by  taking  section*  ot  the 
eurfaoes  of  Figure  3  and  Figure  5  suggeat  that  neither  the  G/D-opcimal  design 
nor  the  3  design  are  universally  superior  one  to  the  other.  In  some 
subregions  one  design  is  slightly  better,  and  in  others  the  other  design  is 
slightly  better.  Both  information  functiono,  and  particularly  that  of  th» 
O/D-optimal  design,  show  a  tendency  to  sag  in  the  middle.  This  happens  for 
the  G/O-optimal  dosign  because  the  O-optimality  oharaoteristio  guarantees  that 
(maximixed)  minima  for  I  ,  aach  equal  to  1/p,  ooour  at  avery  design  poLnt, 

m 

which  must  include  the  osnter  point.  However,  this  sagging  information 

pattern  of  tho  second  order  design  is  not  of  oourse  a  characteristic  of  the 

first  order  design  of  Figure  2  which  is  also  D/Q-optimal  but  contains  no 

osnter  point.  Zf  the  idea  of  tho  desirability  ot  designs  possessing  a 

particular  kind  of  information  profile  is  basic,  then  it  seams  unsatisfactory 

that  the  nature  of  that  profile  should  depend  so  vtry  much  on  ths  order  of  the 

design.  Indeed,  the  relevanoe  of  the  minimax  criterion  which  produces  U- 

optimality  ie  arguable.  Zt  follows  from  the  Klefer-Wolfowits  theorem  that  a 

second  order  design  for  ths  (±1,±1)  region  whose  information  function  did  not 

sag  in  the  middle  would  neoesaarily  not  be  D-optimal.  But  as  we  have  seen  i  D- 

optlmality  is  only  one  of  many  single-valued  criteria  that  might  ba  used  In 

attempts  to  describe  some  important  characteristic  of  ths  X'X  matrix.  Others 

«•  « 

for  example  would  be  A-optimnlity  and  E-optimality,  and  theae  would  yield 
different  information  profiles.  But  I  would  argue  that  sines  the  Information 
function  Itself  is  ths  most  direct  measure  of  desirability  so  far  as  the 
single  issue  of  variance  properties  Is  concerned,  our  best  course  ie  to  choose 
our  design  directly  by  picking  s  suitable  information  function,  and  not 
indirectly  by  finding  soma  extremum  for  A,  E,  D,  or  other  rtu tvary  criterion. 
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figure  6.  Profiles  of 
Ix  for  the  seoond 

order  0/a -op ties 1 
design  end  for  the 
3*  lectori*!  design. 


Sensitivity  of  crlts  rla  to  size  and  shape 

In  the  process  of  scientific  Investigation,  the  investigator  and  the 

statistician  must  do  a  great  deal  of  guesswork.  In  matohing  the  region  of 

interest  R  and  the  degree  of  complexity  of  the  approximating  function,  they 

j  must  try  to  take  into  account,  for  example,  that  a  more  flexible  second  degree 

l 

approximating  polynomial  can  be  expected  to  be  adequate  over  a  larger  region 
K  than  a  first  degree  approximation.  Obviously  different  experimenters  would 
have  different  ideas  of  appropriate  looatione  and  ranges  for  experimental 
variables,  in  particular,  ranges  could  easily  differ  from  one  experimenter  to 

R 

another  by  a  factor  of  two  or  more  .  In  view  of  this,  extreme  sansitivity  of 
design  criteria  to  scaling  is  disturbing^.  For  example,  suppose  each 
dimension  of  a  dth  order  experimental  design  is  increased7  by  s  fsotor  o.  Than 

the  D  criterion  is  increased  by  s  factor  of  wh*r* 


5  Over  a  sequence  of  designs,  initial  bad  ohoioss  of  scale  and  location  would 
tand  to  be  aorreoted,  of  aourse. 

{j  In  particular,  designs  can  only  be  fairly  compared  if  they  are  first  scaled 
to  ba  of  the  "same  site.”  But  how  is  sisa  to  be  measured?  It  was  suggastad 
in  [14]  that  designs  should  be  judged  as  being  of  tha  asms  aiza  when  their 
marginal  second  momenta  E(xiu  -  3L)2/n  were  identical.  This  convention  la 
not  entirely  satisfactory,  but  will  of  course  give  vary  different  results  from 
those  whiah  assume  design  points  to  be  all  included  in  the  same  region  RO.  It 
is  important  to  be  aware  that  the  apparent  superiority  of  ono  design  over 
another  will  often  disappear  if  tha  method  of  scaling  tha  design  ia  changed. 

In  particular  this  applies  to  comparieone  suoh  as  those  made  by  Nallmov  at  al 
[40]  and  Lucas  [ 36 ] . 

7  A  measure  of  efficiency  of  a  daaign  criterion  (see  for  example  [3],  [17])  ip 
motivated  by  considering  the  ratio  of  the  number  of  runs  nsoessary  to  achieve 
tha  optimal  design  to  the  number  of  rune  requirad  for  the  suboptimal  design  to 
obtain  the  same  value  of  the  criterion  (supposing  fractional  numbers  of  runs 
to  be  allowed).  In  particular  for  the  0  criterion,  this  measure  of  In¬ 
efficiency  ia  (U/D  t) ’Z*.  Equivalently  here,  to  illustrate  soale 
seneitivity,  we  concentrate  attention  on  the  factor  c  by  which  each  scale 
would  naad  to  be  inflated  to  achieve  the  same  value  of  the  D  criterion. 


2k(k+d)l 


q  - - 

(kti)lM-Dl 

Equivalently  a  confidence  region  of  the  same  volume  as  that  for  a  D-optimal 

design  can  be  achieved  for  a  design  of  given  D  value  by  increasing  the  scaLo 

for  each  variable  by  a  factor  of  c  ■  (D^^/D)1^,  thus  increasing  the  volume 

k  K  /cj 

occupied  by  the  design  in  the  x  space,  by  a  faotor  c  ■  <Dopt/D>  •  F'or 
example  the  0  value  for  the  32  factorial  design  of  figure  3  is  0.98  y  it)-2  an 
nompared  with  a  D  value  of  1.14  X  10“2  for  the  D-optimal  design.  For  (k  -  2, 
d  •  2),  we  find  q  ■  16,  and  c  •  ( 1  •  1 4/0*98 ) ' / 1  **  -  1.009.  Thus  the  name  value 
of  D  (the  sane  volune  of  a  confidence  region  for  the  S' a)  as  is  obtained  for 
the  D-optimal  design  would  be  obtained  from  a  3*  design  if  each  side  of  the 
square  region  were  increased  by  laaa  than  1%.  Equivalently,  the  area  of  the 
region  would  be  increased  by  leas  than  2%.  Using  the  scaling  that  was  used  in 
Figure  4  for  illustration,  we  should  have  to  uhangw  the  tempura  tor*  by 
20.18  *C  instead  of  20  °C,  and  the  time  by  two  hours  and  one  minute  instoa-1  of 
two  hours,  for  the  32  factorial  to  give  the  same  D  value  as  the  D/o-optimal 
design.  Obviously  no  experimenter  can  guess  to  anything  approaching  this 
accuracy  what  are  suitable  ranges  ovar  which  to  vary  these  factors. 

Obviously  choice  of  region  and  choict  of  information  function  arc 
closely  interlinked.  For  example,  net  of  N  -  k-t-1  points  in  k-spaco  which 
have  no  coplanarities  is  obviously  a  D-optimal  first  order  design  for  some 
ellipsoidal  region.  Furthermore  the  Information  function  for  a  design  of 
order  d  is  a  smooth  function  whose  harmonic  average  over  the  n  experimental 
points  (which  can  presumably  be  regarded  as  rapresantatl ve  of  the  region  of 


a 

Namely  for  that  region  enclosed  withtn  the  information  contour  I  ■»  i/p 
which  must  pars  through  all  the  K+1  experimental  points. 
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interest)  is  always  1/p  wherever  we  place  the  points.  Thus  the  prob1  m  of 
design  ie  not  so  much  a  question  of  choosing  the  design  to  increabe  cotal 
Information  ns  spreading  the  total  information  around  in  the  man.ior  desired 


Rotatable  Designs 

k  route  for  aimplication  different  from  alphabetic  optimality  occurs 
when,  after  suitable  transformation  of  the  inputs  X  to  standardized 
variables  X  nothing  is  known  about  the  orientation  in  the  X  space  of  the 
response  surface  we  wish  to  study.  It  was  argued  by  Box  and  Hunter  [11]  that 

A 

we  should  then  employ  designs  having  the  property  that  the  variance  of  y  is 
a  function  only  of  P  ■  (X' X)^2  so  that 

vx  -  vp  *nd  rx  •  Xp  • 


For  a  first  order  design,  rotatability  implies  orthogonality  and  vies 

versa,  and  coraplstely  dscidst  tha  information  function.  For  second  and  higher 

order  designs,  a  requirement  of  rotatability  fixes  many  moment  properties  of 

the  design,  but  V  and  hence  I  are  still  to  some  extent  at  our  choice,  and 
P  P 

can  be  changed  by  changing  certain  moment  ratios  [11].  In  particular,  for  a 
second  order  design,  vp  depends  on  the  single  moment  ratio 
X  «  (n/3  )EX^/(EX^)^  ,  Por  illustration,  Figure  7  shows  the  information 
function  for  a  second  order  rotatable  design  with  X  ■  .75  consisting  of  B 
points  arranged  in  a  regular  octagon  with  4  points  at  the  center. 

The  truth  seems  to  be  that  at  any  particular  phase  of  an  investigation 
the  scientific  decision  that  must  contributes  to  the  outcome  of  that  phase  it; 
the  choice  of  the  current  region  of  interest  (involving  choice  of  variables, 
locations,  range?,  and  transform*  tiorm )  --  this  ie  .1  choicn  that  dons  not 
reallv  involve  statistics.  After  this  decision  is  made,  (and  given  the 
assumption  that  the  model  fits  perfectly  so  that  only  the  variance  properties 
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Figure  7(a)  Information  function  for  a  second  order  rotatable  design 
consisting  of  8  points  on  a  circle  plus  4  center  points. 


of  th«  design  era  of  interost)  any  set  of  experiments  that  cover  this  region 
in  some  reasonably  uniform  way  is  Likely  to  do  quite  well.  1  c.xnnot  see  that 
the  various  optimality  criteria  are  particularly  relevant  to  this  choice, 
although  there  would  certainly  be  no  harm  in  considering  them,  together  with 
many  other  factors  briefly  discussed  later* 

Ignoring  of  bias 

All  models  are  wrong;  soma  models  are  useful.  This  aphorism  Is 
particularly  true  for  empirical  functions  such  as  polynomials  that  make  no 
claim  to  do  more  then  locally  graduate  the  true  funotion.  For  chemical 
oxamplas  some  idea  of  the  adequaay  of  euch  approximations  can  be  gained  by 

Q 

studying  surfaces  produced  by  chemical  kinetio  models.  An  example  taken  from 
(10]  is  shown  in  Figure  8.  See  also  (15). 

One  conclusion  1  reached  from  many  such  studies  wee  that  approximations 
would  not  need  to  be  very  good  for  response  surface  methods  to  work.  Thus 
within  region  A  of  Figure  8  the  locally  monotonic  function  could  be  crudely 
approximated  by  a  plane  which  could  indicate  a  useful  path  of  ascent.  Also 
valuable  information  might  be  obtained  about  a  ridge  suoh  as  that  in  regLon  B, 
even  though  the  underlying  surface  was  not  exactly  quadratic.  Notice  howeve* 


8  This  surface  was  generated  (see  [10]  for  details)  by  considering  the  yield 

kl  *2 

of  the  product  B  in  a  consecutive  reaction  A — *8— eC  following  first  order 
kinetics  with  temperature  sensitivity  given  by  the  Arrhenius  relation 
In  k^  ■  In  ol  +  Si/ T,  where  temperature  T  is  measured  in  degrees  Kelvin, 
using  plausible  values  for  the  constants  otj,  a 2 ,  6 1 , 
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REACTION  TIME  (HOURS) 


Figure  8.  Contours  of  •  theoretical  response  surface  in  reaction 

time  and  reaction  temperature  for  a  first  order  consecutive 
reaction,  with  plausible  values  substituted  for  kinetic 
constants . 


that  in  the  light  of  such  examples  any  theory  of  experimental  design  which 
depended  on  the  exactness  of  such  approximations  should  be  regarded  with  some 
skepticism. 


S.  TAKING  ACCOUNT  OF  B2AS 

a  as 

If  y  ■  x'0  is  the  fitted  value  using  the  empirical  approximation,  then 

aa  mm 

its  total  error  c  is 

A  a  A  A 

y  -  n  -  {y  -  E(y)}  *■  (E(y)  -  n) 


Thus  the  error  c  contains  a  random  part  *v  and  a  systematic,  or  bias,  part 
c  ,  and  wo  must  expect  that  e  will  not  be  negligible*  Since  all  the  theory 
previously  discussed  makes  the  assumption  that  C  is  saro,  we  must  consider 

D 

whether  the  resulting  designs  are  robust  to  this  kind  of  discrepancy.  The 

optimality  oriteria  discussed  earlier  which  assume  the  response  function  to  be 

exact  usually  produce  a  substantial  proportion  of  sxpsrimsntal  points  on  the 

boundary  of  MO.  In  the  context  of  possible  bias,  this  is  not  reassuring, 

*»  ^ 

since  it  is  at  these  points  that  the  approximating  function  wilL  be  most 
strained. 

The  explicit  recognition  that  bias  will  csrtainly  be  present  does 
however  seem  to  provide  a  more  rational  maans  for  approaching  the  scaling 
problem  (((>],  [7]).  To  see  this,  consider  again  the  formulation  given  earlier 
in  terms  of  n  region  of  interest  R  and  a  larger  region  0  of  operability.  If 

«*  m 

we  were  to  assume  (unrealistically)  that  the  approximation  remained  exact 
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however  widely  the  point*  were  spread ,  and  if  some  measure  of  variance 

reduction  were  the  only  consideration!  then  to  obtain  most  accurate  estimation 

within  R,  the  size  of  the  design  would  have  to  be  increased  to  the  ooundaries 

of  the  operability  region  0.  But  in  fact  of  course  the  wider  the  points  were 

•* 

spread,  the  less  applicable  would  be  the  approximating  function/  and  the 

bigger  the  bias  error.  This  suggests  that  we  should  seek  restriction  of  the 

spread  of  the  experimental  points  not  by  artificial  limitation  to  some  region 

RO,  but  by  balancing  off  the  competing  requirements  of  variance  on  the  one 
«•  « 

hand,  which  is  reduced  as  the  spread  of  the  points  is  increased/  and  bias  on 
the  other  hand,  which  ia  increased  as  the  spread  of  the  points  is  increased. 

A 

The  mean  square  error  associated  with  estimating  by  y^ 

2 

standardized  for  the  number,  n,  of  design  points  and  ths  error  variance  o  , 
can  be  written  as  the  sum  of  e  variance  component  and  a  squared  bias  component 

n»B(ys  -  n^)2/o2  -  n-V(y^)/o2  +  n{E(y^)  -  n^/o2  , 

or 


For  illustration,  an  example  is  taksn  from  a  forthcoming  book  with  N.R. 
Draper  and  J.S.  Hunter  [10}.  Figure  9  shows  a  situation  as  it  might  exist  for 
a  single  variable  when  a  straight  line  approximating  function  is  to  be  used. 
The  diagram  shows  whet  might  be  the  true  underlying  function  which  would  of 
course  be  obscured  by  experimental  error*  Suppose  the  region  of  interest  R 
is  scaled  so  that  -x0  <  x  <  x0  end  in  particular  consider  the  two  designs 

(a)  (-2/3,  0,  2/3)  and  (b)  (-4/3,  0,  4/3). 

Ons  way  (6)  to  obtain  overall  measures  of  variance  and  squared  bias 
over  any  specified  regLon  of  interest  R  is  by  averaging  V  and  B  over 

0»  A  ^ 

«*  «« 

R  to  provide  the  quantities 

m 
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(*) 

Dtcig*  Qoty  • 


Figure  9 


i 


Two  possible  designs  for  fitting  e  straight  line 
over  e  region  of  interest  R  -1  <  x  <  1. 


v  “  and  8  "  JRBxdx//Rdx  • 

Denoting  the  integrated  (over  R)  mean  square  error  by  M,  we  can  then  write 

M  ■  V  +  B. 


Por  the  previous  example,  V,  B,  and  M  are  plotted  against  XQ  in  Figure  10. 

We  see  how  V  becomes  very  large  if  the  spread  of  the  design  is  made  very 
small,  while  if  the  design  is  made  very  large,  V  slowly  approaches  its  minimum 
value  of  unity.  The  average  squared  bias  B,  on  the  other  hand,  has  a  minimum 
value  when  Xq  shout  0,7,  and  increases  for  larger  or  smaller  designs.  A 
rather  flat  minimum  for  M  -  V  ♦  8  occurs  near  XQ  ■  0.79,  Thus  in  this  manner 
the  design  which  minimises  average  mean  squared  error  M  is  not  very  different 
from  the  design  which  minimises  average  squared  bias  B,  but  is  extremely 
different  from  that  wnich  minimises  average  variance  V. 

Choice  of  alternative  model 

A  difficulty  in  all  this  is  that  in  practice  we  do  not  know  the  nature 

true  function  n^.  Progress  may  ba  made  however  by  supposing  that 
nX  is  t.  some  satisfactory  approximation  raprasented  by  a  polynomial  model  of 
higher  degree  dr  Suppose  then  thst  a  polynomial  model  of  degree  d,  is  fitted 
to  n  data  values  to  give 

-X  "  *1-1 

while  the  true  model  is  in  fact  a  polynomial  of  degree  d2,  so  that 


3*  +  *a!a 


We  also  need  to  know  something  about  the  relative  magnitudes  of 
systematic  and  random  errors  that  we  could  expect  to  meet  in  practical 
cAses.  It  was  argued  in  [6]  that  an  investigator  might  typically  employ  a 
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DesKN(b) 


“*“*  10'  °!  int*»r*“a  v.rl«c.  v.  Integrated 

aquarad  bit  a  B,  and  thair  total  m  »  v  +  b,  for 
thraa-point  daaigna  (-X0,0,XQ0). 
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fitted  approximating  function  such  as  a  straight  line  when  he  believed  that 
the  average  departure  from  the  truth  induced  by  this  approximating  function 
was  no  worse  than  that  induced  by  the  procoas  of  fitting.  This  would  suggest 
that  the  experimenter  would  tend  to  choose  the  sise  of  his  region  R,  and  the 
degree  of  his  approximating  function  in  such  a  way  that  the  integrated  random 
error  and  the  integrated  systematic  error  were  about  equal.  Thus  we  might 
suppose  that  a  situation  of  particular  interest  is  that  where  B  is  roughly 
equal  to  V.  Kxamples  that  we  studied  seamed  to  show  that  designs  that 
minimised  H  with  the  constraint  V  ■  B  were  close  to  those  which  minimised  B. 
Consequently  we  suggested  that,  if  a  simplification  were  bo  be  made  in  the 
design  problem,  it  might  almost  be  better  to  ignore  the  effeots  of  sampling 
variation  rather  than  those  of  bias. 

However  this  may  be,  there  seems  no  doubt  that,  in  making  a  table  of 
useful  designs,  a  component  in  our  thinking  should  be  the  characteristic*  of 
the  designs  which  minimised  equated  bias  against  feared  alternatives.  As  a 
factor  in  our  final  choice,  this  should  oertainly  receive  as  much  attention  as 
the  indications  supplied  by,  say,  D-optimality. 

for  illustration  particular  examples  of  designs  in  three  dimensions 
which  minimise  integrated  squared  bias  when  R  is  a  sphere  of  unit  radius  are 
shown  in  Figure  11(a)  for  d^  ■  1  and  d2  ■  2  (a  first  order  design  robust  to 
second  order  effeots)  and  in  Figure  11(b)  for  dj  ■  2  and  d2  ■  3  (a  second 
order  design  robust  to  third  order  effeots).  The  former  is  the  familiar  23 
factorial  scaled  so  that  the  points  are  0,71  units  from  the  center.  The 
latter  is  a  rotatable  composite  design  with  "cube"  points  at  a  distance  0.86 
from  the  center,  and  "star"  points  at  a  distance  0,83  from  the  center. 
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Figure  11(a)  A  first  order  (two-level  factorial) 

design  in  three  factors  which  niniaieas 
squared  bias  from  second  order  terns 
when  the  region  of  interest  is  a  sphere 
of  unit  radius. 


Figura  11(b)  A  aacond  ordar  compoaita  rotatabla  daaign  which 
niniaisaa  aquarad  biaa  front  third  ordar  tarma 
whan  tha  waight  function  ia  uniform  ovar  a 
apharical  ragion  of  intaraat  of  unit  radiua. 


Figure  12 t  Two  possible  weight  funotions  for  k  *  It 

(e)  "Uniform  over  R"  type  indiaeting  uniform 
interest  over  R,  no  Interest  outside  Ri 
<b)  Normal  Distribution  shape,  giving  greeter 
weight  to  points  nearer  F. 


Obviously  in  practice  because  of  the  inevitable  inexactness  of 

choosing  scales  exact  dimensions  of  the  designs  should  not  be  taken  too 

seriously,  but  these  examples  illustrate  the  fact  that  as  30on  as  we  take 

account  of  bias,  design  points  are  not  chosen  on  the  boundary  of  R. 

Choice  of  designs  which  minimize  piaa 

Before  considering  the  problem  of  choosing  minimum  bias  designs  it  is 

desirable  to  generalize  slightly  the  previous  formulation.  Although  it  avoids 

limiting  the  location  of  the  design  points  in  an  artificial  way  the  idea  of  a 

region  of  interest  R  within  a  larger  operability  region  0  ie  still  not 

entirely  satisfactory  because  it  implies  that  we  have  equal  interest  at  all 

points  within  R.  A  more  general  formulation  (7]  which  subsumes  that  we  have 

been  discussing  employs  a  weight  function  w(x)  which  extends  over  the 

operability  region  O  so  that  /  w(x)dx  ■  i.  The  weighted  mean  square  error  M 

•» 

can  now  be  split  into  a  weighted  variance  part  V  and  a  weighted  squared  biaB 
part  B  so  that  again  M  «  V  +  B,  with 

M  -  /_w(x)s{yU)  -  nU)}2dx 

v  •*  •»  ■*  e 

m 

a  a  n 

V  -  /  w(X)E{y{x)  -  E(y( x) )}  dx 

w  m  m  e  « 

B  -  /QwU){E(yU)  -  n(x)}2dx  . 

Two  possible  weight  functions  for  k  ■  1  120)  are  shown  in  Figure  12. 

Suppose  as  before  the  fitted  function  is  a  polynomial  X.jb^  of 
degree  d1  while  the  true  model  is  a  polynomial  +  x2®2  °*  de9cee  d2 

and  define  moment  matrices  for  the  design  and  for  the  weight  function  by 
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">2  ■ 


s.i  •  “.2  ■ 


Then  (6)  a  necessary  and  sufficient  condition  for  the  squared  bias  B  to  be 
minimized  is  that 


!il?u  '  “tliia 


and  hence  a  sufficient  condition  is  that  all  the  moments  of  the  design  up  to 
and  including  order  dj  +  dj»  ere  equal  to  all  the  corresponding  moments  of  the 
weight  function. 


6.  SOME  OTHER  CONSIDERATIONS  IN  DESIGN  CHOICE 


There  is  insufficient  space  to  discuBS  here  all  of  the  items  in  Table  1 
that,  in  one  cvrcumstance  or  another,  it  might  he  necessary  to  take  into 
account,  but  mention  will  be  made  of  a  few. 

Lack  of  Fit  (ill).  Sequential  Assembly  (vl),  Blocking  (v),  Estimation 
of  Error  (vli),  Transformation  Estimation  (iv) 

while  the  adequacy  of  a  particular  approximating  function  to  explore  a 
region  of  current  interest  is  always  to  some  extent  a  matter  of  guesswork, 
simple  approximations  requiring  fewer  runs  for  their  elucidation  will  usually 
be  preferred  to  more  complicated  ones.  This  leads  to  a  strategy  of  building 
up  from  simpler  models,  rather  than  down  from  more  complicated  ones.  A 
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practical  procedure  ia  then:  to  employ  the  simplest  approximating  function 

which  it  is  hoped  may  be  adequate!  to  allow  for  checking  its  adequacy  of  fit 

(see  also  (1),  [2],  (6),  and  119])}  to  switch  to  a  more  elaborate  * 

approximating  function  when  this  appears  necessary.  The  implication  for 

designs  is  (a)  that  they  should  provide  for  checking  model  adequacy,  (b)  that 

i 

they  should  be  capable  of  sequential  assembly  —  a  design  of  order  d  should  be 
augmentable  to  one  of  order  d  +  1,  (c)  since  conditions  may  change  slightly 
from  one  aet  of  runs  to  another,  especially  affecting  level,  the  pieces  of  the 
design  should  form  orthogonal  blocks. 

For  illustration,  Figure  13  shows  the  sequential  assembly  of  a  design 
arranged  in  three  orthogonal  blocks,  each  of  six  runs,  labeled  I,  II,  and  III. 

Block  I  is  a  first  ordsr  design  but  also  provides  a  check  for  overall 
curvature  (obtained  by  contrasting  the  average  response  of  the  center  points 
with  the  average  response  on  the  cube).  A  single  contrast  of  the  center 
response  is  available  as  a  gross  chsck  on  previous  information  about 
ex  lmental  error.  If  after  analysing  the  results  from  Block  I  there  are 
doubts  about  the  adequacy  uf  a  first  degree  polynomial  model,  Block  II  may  be 
performed.  It  uses  the  complementary  simplex,  and  the  two  parts  together  form 
a  first  order  design  (I+II)  with  much  greater  ability  to  detect  lack  of  fit 
due  to  second  ordsr  teems  provided  by  additional  orthogonal  contrasts 
estimating  the  two-factor  interactions.  The  addition  of  Block  III  produces  a 
composite  design  (I+II+III)  whicn  allows  a  full  second  degree  approximating 
equation  to  be  fitted  if  this  appears  to  be  desirable.  The  complete  design 
also  provides  orthogonal  checking  contrasts  for  lack  of  quadraticity  in  each 
of  the  three  directions  ((9),  (12)).  These  contrasts  can  also  be  regarded  as 
checking  the  need  for  transformation  in  each  of  the  X's.  Finally  if  it  were 
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decided  that  more  information  about  experimental  error  was  desirable,  the 
replication  of  the  star  in  a  further  Block  IV  could  furnish  this,  and  also 
provide  some  increase  in  the  robustness  of  the  design  to  wild  observations. 

Robustness 


Approaches  to  the  robust  design  of  experiments  have  been  recently 
reviewed  by  Herzberg  [28];  see  also  [29].  In  particular.  Box  and  Draper  [8] 
suggested  that  the  effects  of  wild  observations  could  be  minimised  by  making 
r  “  small,  where  R  ■  {r  }  ■  X(X'X)  ^X*.  This  is  equivalent  to 

UU  «  lU  m  «  m  m 

2  2  A 

minimizing  £ruu  -  p  /n  «  Var(V{y)}  which  takes  the  value  zero  when 

a 

v(y  )  -  p/n  (u  -  1,2,...,n).  Thus  G-optimal  designs  are  optimally  robust  in 
this  sense. 

Size  of  the  experimental  design 

A  good  experimental  design  is  one  which  focuses  experimental  effort  on 
what  is  judged  important  in  the  particular  current  experimental  context. 
Suppose  that,  in  addition  to  estimating  the  p  parameters  of  the  assumed 
model  form,  it  is  concluded  that  f  >  0  contrasts  are  needed  to  check 
adequacy  of  fit,  b  >  0  further  contrasts  for  blocking,  and  that  an  estimate 
of  experimental  error  is  needed  having  e  >  0  degrees  of  freedom.  To  obtain 
independent  estimates  of  all  items  of  interest  we  then  require  a  design 
containing  at  least  p  +  f  +  b  +  e  runs.  However  the  importance  of  checking 
fit,  blocking,  and  obtaining  an  independent  estimate  of  error  will  differ  in 
different  circumstances,  and  the  minimum  value  of  n  will  thus  correspondingly 
differ.  But  this  minimum  design  will  in  any  case  only  he  adequate  if  0  Is 
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below  bob®  critical  value.  When  0  is  larger  designs  larger  than  the 


i 

minimum  design  will  be  needed  to  obtain  estimates  of  sufficient  precision.  Zn 
this  circumstance  rather  than  merely  replicate  the  minimum  design,  opportunity 
may  be  taken  to  employ  a  higher  order  design  allowing  the  fitting  of  a  more 
elaborate  approximating  function  which  can  then  cover  a  wider  experimental 
region*  Notice  that  even  when  a"  is  small  designs  for  which  n  is  larger 
than  p  are  not  necessarily  wasteful*  This  depends  on  whether  the  additional 
degrees  of  freedom  are  genuinely  used  to  achieve  the  experimenter's  current 
objectives. 

Simple  Pete  Patterns 

Xt  has  sometimes  been  argued  that  we  may  as  wall  choose  points  randomly 
to  cover  the  "design  rsgion"  or  employ  some  algorithm  that  distributes  them 
evenly  even  though  this  does  not  rasult  in  a  simple  data  pattern  euch  as  is 
achieved  by  factorials  and  composite  response  surface  designs*  In  favor  of 
this  idea  it  has  been  urged  that  the  fitting  of  a  function  by  least  squares  to 
a  haphazard  set  of  points  is  no  longer  a  problem  for  modern  computational 
devices*  This  is  true*  but  overlooks  an  important  attribute  of  designs  which 
form  simple  patterns.  The  etatiatician's  task  as  a  member  of  a  scientific 
team  is  a  dual  one,  involving  inductive  criticism  and  deductiva  estimation. 

The  latter  involves  deducing  in  the  light  of  the  data  the  consequences  of 
given  assumptions  (estimating  the  fitted  function),  and  title  can  certainly  be 
dona  with  haphazard  designs.  But  tits  former  involves  the  question  (a)  of  what 
function  should  be  fitted  in  the  first  place,  and  (b)  of  how  to  examine 


280 


residuals  from  the  fitted  function  in  an  attempt  to  understand  deviations  from 


the  initial  model,  in  particular  in  relation  to  the  independent  variables,  and 
so  to  be  led  to  appropriate  model  modification. 

Designs  such  as  factorials  and  composite  response  surface  designs 
employ  patterns  of  experimental  points  that  allow  many  such  comparisons  to  be 
made,  both  for  the  original  observations  and  for  the  residuals  from  any  fitted 
function,  l-'or  example,  consider  a  3  factorial  design  used  to  elucidate  the 
effects  of  temperature  and  concentra tion  on  some  response  such  as  yield. 
Intelligent  inductive  criticism  is  greatly  enhanced  by  the  possibility  of 
being  able  to  plot  the  original  data  and  residuals  against  temperature  for 
each  individual  level  of  concentration,  and  against  concentration  for  each 
individual  level  of  temperature. 


7.  CONCLUSION 

(i)  We  must  look  for  good  design  criteria  which  measure 
characteristics  of  the  experimental  arrangement  in  which  the  scientist  might 
sensiDly  be  interested.  Because  the  in*i  ortance  of  various  characteristics 
will  differ  in  different  situations,  tables  of  such  criteria  for  particular 
designs  would  encourage  good  judgment  to  be  used  in  matching  the  design  to  the 
scientific  context.  Optimum  levels  of  these  criteria  can  be  useful  as  bench 
marks  in  judging  the  efficiencies  of  a  particular  design  with  respect  to  these 
various  criteria. 

(ii)  However  good  designs  must  in  practice  be  good  compromises,  and  it 
is  doubtful  how  useful  single  criterion  optimal  designs  are  in  locating  such 
compromises.  An  optimal  design  is  represented  by  a  point  in  the  multi 
dimensional  space  of  the  coordinates  of  the  design  and  a  series  of  different 
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criteria  will  give  a  series  of  such  extremal  points  which  can  be  very 
differently  located.  Obviously  knowledge  of  the  location  of  such  extrema  may 
tell  us  almost  nothing  about  the  location  of  good  compromises.  For  this  we 
would  need  to  study  the  joint  behaviour  of  the  criterion  functions  at  levels 
close  to  their  extremal  values.  One  limited  but  useful  step  would  be  to 
further  investigate  which  criteria  are  in  accord,  (such  as  G-optimality  and 
robustness  to  wild  observations)  and  which  in  conflict  (such  as  variance  and 
bias ). 

(iti)  It  is  true  that  the  problem  of  experimental  design  is  full  of 
scientific  arbitrariness  --  no  two  investigators  would  choose  the  same 
variables,  start  their  experiments  in  the  same  place,  change  variables  over 
the  same  regions,  and  so  on  —  but  science  works  not  by  uniqueness  but  by 
employing  iterative  techniques  which  tend  to  converge.  Clearly  we  must  learn 
to  live  with  scientific  arbitrariness,  or  slae  we  are  in  a  world  of  make 
believe.  But  we  can  make  the  problems  worse,  not  better,  by  introducing 
arbitrariness  for  purely  mathematical  reasons. 
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ABSTRACT 

'3  In  the  formal  study  of  design  and  analysis  of  experiments,  it  is  often  overlooked  that  a 
simple  and  straight-forward  design  can  become  complicated  during  analysis.  Presented  here  is  a 
specific  case  in  which  the  design  was  readily  apparent  but  where  difficulties  subsequently  arose. 
Analysis,  plagued  by  nonhomogeneity  of  variance  and  the  suspicion  of  a  lurking  variable,  is  dis¬ 
cussed^ 

INTRODUCTION 

Answers  to  questions  concerning  the  performance  of  a  MLRS  (Multiple  Launcher  Rocket 
System)  bomblet  were  desired  The  M42  is  a  small  shaped-charge  bomblet'Tfigure  1);  designed 
to  detonate  on  impact  causing  a  jet,  comprised  primarily  of  copper,  to  penetrate  the  armor 
which  it  has  impacted  Many  bomblets  are  placed  within  a  time-fused  rocket,  which  is  flown 
over  the  target  area.  A  charge  within  the  rocket  is  ignited,  causing  the  skin  of  the  rocket  to  peel 
away.  This  allows  the  undetonated  bomblets  to  be  sprayed  over  the  target  area;  as  the  bomblets 
fall  to  the  ground,  a  portion  of  them  will  impact  the  target.  - - - — - - - - 

DESIGN 

There  were  three  questions  about  the  performance  of  this  munition  to  be  answered 
First,  is  there  a  difference  in  bomblet  performance  among  vendors?1  Tn  this  study,  performance 
of  the  bomblet  was  taken  to  be  penetration  depth  of  the  jet  into  the  target  This  question  is 
self-explanatory  and  we  will  only  note  that  there  were  three  vendors  considered  Second,  does 
the  dispersing  process  have  an  effect  on  bomblet  performance?  Dispersing  is  the  process  by 
which  the  bomblets  are  delivered  from  the  rocket  to  the  target  Tn  particular,  the  customer  was 
concerned  with  the  ignition  of  the  charge  within  the  rocket  which  causes  the  skin  of  the  rocket 
to  peel  away.  When  this  charge  is  ignited,  the  bomblets  are  subjected  to  a  certain  amount  of 
force.  The  above  question  then  becomes  how  does  this  force  affect  bomblet  performance.  Tn 
order  to  answer  this  question,  one  half  of  the  bomblets  went  through  the  dispersing  simulation 
before  testing  for  penetration  depth.  Third,  how  does  Standoff  affect  bomblet  performance? 
Standoff  is  the  distance  above  the  target  at  which  the  bomblet  is  detonated  The  customer  was 
interested  in  bomblet  performance  where  detonation  occurs  at  four  different  heights  above  the 
target. 

To  answer  these  questions,  an  experimental  design  was  developed  (figure  2).  A  2x3x4 
factorial  design  with  response,  Penetration  Depth,  and  with  factors,  Dispensing,  Vendor,  and 
Standoff  was  chosen.  Tn  consideration  of  available  bomblets,  six  observations  per  cell  were 
used  This  design  was  then  suggested  to  the  customer  who  then  contracted  a  third  party  to  run 
the  experiment. 
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ANALYSIS 

In  examining  the  data,  irregularities  in  the  values  caused  us  concern  with  respect  to  the 
usual  model  assumptions  of  normality,  homogeneity  of  variance,  and  additivity.  Prior  to  per¬ 
forming  an  analysis  of  variance,  testing  of  those  assumptions  was  begun.  To  test  for  normality, 
a  Shapiro- Wilk  test  was  run  on  the  observations  within  cells.  At  the  .05  significance  level  we 
found  the  results  not  inconsistent  with  the  assumption  of  normality.  Turning  then  to  the  ques¬ 
tion  of  homogeneity  of  variance,  a  plot  (figure  3)  of  the  cell  means  against  the  ceil  variances 
was  constructed.  When  examining  this  graph,  it  was  fairly  obvious  that  conditions  were 
somewhat  less  than  ideal.  Various  corrective  measures  using  the  common  transformations  were 
unsuccessful  in  obtaining  homogeneity  of  variance.  Thus,  efforts  were  begun  to  determine  the 
cause  of  heterogeneity  of  variance. 

A  more  critical  look  at  the  data  revealed  that  within  many  of  the  ceils  representing  disper¬ 
sed  bomblets  there  seemed  to  be  two  populations  of  data,  a  group  of  high  values  and  a  group  of 
low  values.  Subjectively  we  flagged  the  lower  values.  Graphically  (figure  4)  we  compared  the 
means  of  the  lower  values  and  the  means  of  the  higher  values  within  a  given  ceil.  On  the  plot, 
the  symbol  at  the  approximate  coordinates  (.75, .75)  represents  the  mean  of  the  lower  values 
from  vendor  1  at  the  first  standoff.  Noting  the  obvious  difference  between  the  mean  of  the 
lower  and  upper  values  within  a  given  cell,  we  began  to  feel  that  maybe  there  were  in  fact  two 
populations  of  data.  It  was  at  this  point  that  we  first  suspected  the  existence  of  a  lurking  varia¬ 
ble. 


In  mid-stream  we  were  asked  to  look  at  the  effect  of  a  new  variable,  Damage,  which  is  a 
measure  of  ’out  of  round’  of  the  bomblet.  It  was  previously  conjectured  that  the  disperse 
process  may  affect  bomblet  performance.  Damage  was  an  attempt  at  a  more  precise  explanation 
of  the  possible  effect  of  dispersing.  In  explaining  how  this  mea: 'irement  was  taken,  it  is  neces¬ 
sary  that  the  testing  sequence  and  apparatus  first  be  described.  First,  bomblets  are  disarmed 
and,  noting  each  bomblet  position,  loaded  into  a  rocket-like  canister  comprised  of  five 
bomblet- holding  packs  (figure  5).  The  dispersing  simulation  involves  exploding  a  charge  within 
the  canister  causing  bomblets  to  be  sprayed  over  the  test  area.  The  bomblets  are  then  gathered 
and  measured  for  Damage,  which  is  the  absolute  difference  of  two  perpendicular  measurements 
of  bomblet  diameter.  After  this  simulation,  the  bomblets  are  armed  and  detonated  at  various 
heights  over  a  plate  of  armor  for  the  penetration  depth  data.  Looking  at  this  variable,  Damage, 
led  us  to  find  our  lurking  variable. 

Investigation  of  Damage  brought  out  the  following  observations.  First,  those  bomblets 
positioned  in  packs  one  and  two  during  the  dispersing  simulation  sustained  a  higher  level  of 
Damage  than  did  those  positioned  in  packs  three  through  five.  Second,  those  bomblets  posi¬ 
tioned  in  packs  one  and  two  during  the  dispersing  simulation  showed  poorer  penetration  than 
did  those  in  packs  three  through  five.  Third,  high  levels  of  Damage  sustained  by  the  bomblets 
adversely  affected  penetration  performance.  These  observations  are  supported  graphically  by 
representative  figures  6  and  7. 

In  figure  6,  the  symbol  at  the  approximate  coordinates  (l.,3.75)  represents  the  mean 
Damage  sustained  by  bomblets  positioned  in  pack  one  during  the  dispersing  simulation  and 
then  fired  at  the  7.72  inch  standoff.  The  symbol  at  the  approximate  coordinates  (l.,l.) 
represents  the  mean  penetration  depth  achieved  by  those  same  bombieta.  Note  that  in  each 
graph  the  highest  level  of  Damage  is  sustained  by  bomblets  from  pack  one  and  that  the  level  of 
damage  decreases  for  bomblets  from  higher  packs.  Also  the  lowest  mean  penetration  depth  is 
exhibited  by  bomblets  from  pack  one  and  generally  Increases  for  bomblets  from  higher  packs. 
The  apparent  relationship  between  Damage  and  Penetration  Depth  was  important,  but  not 
totally  unexpected.  More  interesting  and  more  important  was  the  relationship  of  Pack  to  both 
Damage  and  Penetration  Depth. 
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At  one  point  ear'.y  ip  the  analysis  we  flailed  bomblets  showing  lower  penetration  depths 
u  poMibly  coming  from  a  different  population.  The  relationahip  between  Pack  and  ihoae 
penetration  dept  ha  being  flagged  ia  pointed  out  further  in  figure  8.  Of  fourteen  bombleta  poai- 
doned  in  pack  one  during  the  dispersing  simulation,  eleven  were  flagged  for  low  penetration 
Of  fifteen  bombleta  positioned  in  pack  two,  nine  were  flagged  for  low  penetration  Finally,  of 
twenty  seven  bombleta  flagged,  twenty  had  been  positioned  in  packs  one  or  two  during  the 
dispersing  simulation  Due  to  its  unexpected  effect  on  bombiet  performance.  Pack  wss  deter¬ 
mined  to  be  our  lurking  variable. 

Why  did  Pack  have  an  effect  on  penitmtion  depth?  One  possibility  waa  proposed  by  a 
systems  analyst  familiar  with  MLRS  munition*.  In  figure  5,  note  that  steel  places  were  bolted 
to  the  top  of  pack  five  and  to  tha  bottom  of  pack  one.  Rather  than  being  suapendad  in  air,  the 
test  apparatus  rested  on  the  ground  Whan  tha  charge  within  tha  cat  lister  was  ignited,  the  shell 
of  the  canister,  the  bombleta,  and  the  steel  plate  on  pack  five  were  blown  out  away  from  the 
center  of  the  canister.  The  bottom  steel  plate  remained  stationary,  pinned  by  the  fbrae  of  the 
explosion  and  the  ground.  Many  bombleta  from  the  lower  packs  caromed  off  this  hard  fixed 
surface,  causing  more  severe  deformation  to  themselves. 

CONCLUSION 

In  conclusion,  some  information,  not  add rested  here,  could  still  be  extracted  from  these 
experimental  data,  but  problems  created  by  the  lurking  variable  hindered  the  intended  complete 
analysla.  It  i>  interesting  to  note  that  heterogeneity  of  variance  played  a  hero's  role  in  this 
analysis,  since  investigation  of  this  problem  aided  in  the  discovery  of  the  lurking  variable.  Pack. 
Also,  proper  design  made  it  possible  to  draw  some  conciusiona  in  the  face  of  unexpected  cir¬ 
cumstances.  Finally,  as  suggested  by  Professor  O.B.P.  Box  during  this  presentation,  this  exam¬ 
ple  illustrates  that  statistical  analysis  can  accomplish  much  more  than  hypothesis  testing  by  len- 
ding  insight  to  the  physical  environment,  in  this  case  by  poindng  out  possible  Inadequacies  in 
the  test  apparatus. 
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ABSTRACT.  This  paper  addresses  the  development  of  a  time  series  model  to 
forecast  the  central  procurement  workload  in  the  U.S.  Army  Materiel  Development 
and  Readiness  Cormiand  (DARCOM).  The  Box-Jenkins  approach  of  Identification, 
Estimation  and  Diagnostic  Checking  is  used  to  build  a  seasonal  auto-regressive 
integrated  moving  average  model  (SARIMA)  to  forecast  quarterly  procurement 
actions  (QPA) ,  the  procurement  workload  indicator.  Models  are  developed  using 
60  and  61  data  points  from  FY  65  through  FY  79  to  include  7T.  Forecasted  values 
cure  compared  to  actual  values  for  FY  80,  81  and  82.  ^ _ _ _ _ _ 

I.  Introduction.  The  user  of  the  Box- Jenkins  methodology  for  time  series 

forecasting  is  required  to  exercise  judgment  in  the  choice  of  a  particular 


model  from  a  general  class  of  autoregressive  integrated  moving  average  ( ARIMA)  models 
d_ct lB  _  $2b2  _  _  $  bP)  (1-B)d  Xt  = 

(1_0]LB  -  02b2  -  ...  -  OgB^)  et 

where  et  has  mean  zero,  fixed  variance  and  et  and  ^u  uncorrelated  for  t^u 


and  B  is  the  backshift  operator  defined  as  B^j.  =  X^.^. 

For  seasonal  time  series  of  period  s  the  model  is  generalized  to 


(l-*lB  -  ♦  2b2  "  •••  "  4>pBP)  (1-4>-j  s  ES  -  ...  -  $psfSBPSs){l-B)d(l-Bs)dsXt 
=  (l-O^B  ...  OqB  )d”GifSB  -  ...  ~  Oq^ f sB^sS)  et 

The  task  is  to  select  a  specific  model  from  the  general  class  by  choosing 
appropriate  values  for  p,  d,  and  q.  These  values  are  determined  by  examining 
the  sample  autocorrelations  and  partial  autocorrelations.  The  coefficients  <J> 
and  oare  estimated  by  a  nonlinear  optimization  algorithm  and  the  residuals  of 
the  proper  fitted  model  should  resemble  the  properties  of  a  white  noise  process. 

The  computer  program  usee]  for  determining  the  optimal  parameter  values  is  a 
FORTRAN  program  called  ERSF,  Estimation  of  Rational  Distributed  Log  Structural 
Form  Mrxlcls,  developed  by  K.  D.  Wall,  School  of  Engineering,  University  of  Virginia. 
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II.  Model  Develofinent. 

Models  are  developed  using  60  and  61  data  points  from  FY  65  through  FY  79 
to  include  7T.  One  model  is  fitted  for  60  points  ary)  two  for  61  points.  The 
oorrelograin  of  both  series  indicate  non-stationar ity  and  seasonality  as  shown 

in  figure  i. 
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Figure  1.  Autocorrelations  of  Original  Data 


Forecasted  values  are  compared  to  actual  values  for  FY  00,  81,  and  82. 
The  mean  absolute  percentage  error  (MAPE)  is  used  as  a  measure  of  forecast 
accuracy  rather  than  the  mean  squared  error  (MSE).  The  MAPE  has  a  more  intuitive 
interpretation  than  the  MSE,  and  hence  more  understandable  to  non-statisticians. 
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A.  Model  I 


The  first  model  aridressed  is  fitted  to  60  data  points.  When  a  seasonal 
fourth  difference  is  taken  the  correlagram  in  figure  2  results,  sliowing  two 
autocorrelations  significantly  different  from  zero,  r^  and  An  MA(2),  ARMA 
(1,1)  and  AR< 1 )  were  then  fitted  with  only  the  residuals  of  the  AR(1)  exhibiting 
tlie  characteristics  of  a  white  noise  process.  The  autocorrelation  of  the 
residuals  of  the  following  model 

(1-B4)  (1-0.67205B)  Xt  « 

is  shewn  in  figure  3.  These  appear  to  be  random  and  hence  possibly  a  correct  model  j 
far  forecasting  purposes. 
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Autocorrelations  o£  Seasonal  Fourth  Difference 
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B.  Model  II 

The  7T  data  point  (the  quarter  which  occurred  when  the  fiscal  year  was 
changed  from  1  July  -  30  June  to  1  Oct  -  30  Sept)  was  added  to  the  time  series 
and  a  new  model  was  fitted.  A  first  difference  was  taken  and  a  seasonal 
parameter  added  resulting  in  the  correlogram  of  figure  4.  The  pattern  of  the 
autocorrelations  is  that  of  an  MA(3).  The  autocorrelations  of  the  residuals  of 
the  model 

( 1-B) (1-0. 51982B4)  Xt  =  (1 .0.2895B-0.13664B2  -  0.60893B3)  Efc 
are  shown  in  figure  5.  Although  the  autocorrelations  of  the  residuals  are  all 
within  one  standard  error  a  trend  still  appears  to  exist  since  rj  through  rjj 
are  all  positive.  This  trend  caused  9ome  doubt  as  to  whether  the  residuals 
were  truly  random,  and  so  a  third  model  was  attempted. 


302 


I. 

Ml 

H- 

0. 

Ml 

He 

1. 

Ml 

He 

0. 

MO 

He 

0. 

3  03 

Me 

c* 

f  SO 

He 

•* 

roe 

H* 

It* 

N* 

H* 

It* 

N* 

H* 

M* 

It* 

M* 

It* 

H* 

........ 

HI  • 

Ml  • 

Ml 

Ml 

It*  ...... 

Ml 

III 

MI 

Ml 

MI 

Ml 

MI 

Ml 

MI 

Ml 

Ml 

MI 

Ml 

Ml 

H  * 


Figure  4. 


Autocorrelations  of  First  Difference  and  Seasonal  Parameter 
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Figure  5.  Autocorrelations  of  Itesiduals  of  Model  2 
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Model  111 


f' 


In  an  attempt  to  eliminate  the  trend  of  model  2  a  fourth  difference 
instead  of  the  first  difference'  is  taken  but  the  seasonal  parameter  along  with 
the  three  moving  average  terms  are  kept*  This  resulted  in  the  correlogram  of 
figure  6.  When  a  fourth  moving  average  term  is  added,  the  autocorrelations  of 
the  residuals  exhibit  a  random  pattern  as  shown  in  figure  7.  The  resulting 
model  is 

(1-B4)(1-0.6016234)  Xfc  -  (1+0.81815B  +  0.87187B2 

+  0.66825B3  -  0.29190B4)  efc 


III.  Conclusion 

The  three  models  along  with  their  forecasts  are  compered  to  the  actual 
values  for  fiscal  years  1980,  1981  and  1982,  in  tables  1,  2  and  3.  vhe  mean 
absolute  percentage  error  for  model  2  is  the  lowest.  The  forecasts  from  model 
2  were  choeen  to  predict  future  workload  back  in  fiscal  year  1979. 


I  alt  I  ssses— ee  >«esSsssae  mm  mm 


*.u« 

*• 

Mil 

i.n 

Ha 

*.rt« 

H* 

t.rt « 

Mft 

s.sss 

H# 

s.sic 

1*  • 

Mt( 

M* 

t.llt 

l.tll 

*• 

I.Ut 

*• 

l.llf 

h* 

•  •ISO 

H* 

•  .MO 

H* 

•  .ISO 

H* 

•  .!•« 

H  • 

• 

A  H 

M 

•  • 

• 

• 

a 

• 

a  a  M 

••.too 

Ml 

*  •  »  S  #8 

a  a 

a  a  a  an 

Ml 

as  * 

a  a 

a  #M 

•(.lit 

Ml 

t>  a 

aN 

*«.(•• 

Ml 

a 

etf 

••.IS# 

Ml 

lit 

-t.MO 

Mt 

-•.♦SO 

Ml 

•s.s«o 

Mt 

•s.sse 

Mt 

•(.lit 

Ml 

•S.ilO 

Mt 

•1.1*1 

Ml 

-•.MS 

Mt 

Ht 

-S.ISS 

Ml 

•I.S00 

Mt 

■I.H1 

Ml 

* 

“1*00$  * 


Figure  6.  Autocorrelations  of  Seasonal  Fourth  Difference  of  Model  2 
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Table  1 


Performance  of  Model  1 
MODEL  1  (1-B4)  (1-0.672058)  Yt  =  efc 


FY*80  FY *  81 


FORECAST 

ACTUAL 

PE 

FORECAST 

ACTUAL 

PF 

1st  Qtr 

23,322 

27,455 

15.05 

22,784 

29,455 

22.65 

2nd  Qtr 

35,210 

35,034 

.50 

35,572 

34,938 

1.81 

3rd  Qtr 

40,198 

36,864 

9.04 

39,955 

37,182 

7.46 

4th  Qtr 

40,721 

36,562 

11.38 

40,884 

30,802 

5.36 

TOTAL 

139,451 

135,915 

139,195 

140,377 

FY'82 


FORECAST 

ACTUAL 

PE 

1st  Qtr 

22,674 

30,330 

25.24 

2nd  Qtr 

35,646 

35,040 

1.73 

3rd  Qtr 

39,905 

41,282 

3.34 

4th  Qtr 

40,918 

45,739 

10,54 

TOTAL 

139,143 

153,194 

MAPE  «  9i50 


306 


Table  2 


Performance  of  Model  2 


MODEL  2  U-B)  (1-0.51982B4)  Yfc“ <1.0.28958  -  0.13664B2  -  0.60893B3) 


F’Y'UO 


FORECAST 

ACTUAL 

1st  Qtr 

32,304 

27,455 

2nd  Qtr 

32,854 

35,034 

3rd  Qtr 

39,199 

36,864 

4th  Qtr 

38,435 

36,562 

TOTAL 

142,852 

135,915 

1st  Qtr 
2nd  Qtr 
3rd  Qtr 
4th  Qtr 


FY'81 


PE 

FORECAST 

ACTUAL 

17.88 

34,507 

29,455 

G.22 

34,762 

34,938 

6.33 

38,060 

37,182 

5.12 

37,663 

38,802 

144,992 

140,377 

F7’82 

FORECAST 

ACTUAL 

PE 

35,621 

30,330 

17.44 

35,753 

35,040 

2.03 

37,468 

41,282 

9.24 

37,261 

45,739 

18.54 

e 


17 

0 

2 

2 


TOT  AT. 


146,103 


153,194 


Table  3 

Performance 

of  Model  3 

V 

t 

MODEL  3 

(1-ir1)  (1-0.60I62P4)  Yt  =  (1  + 

0.81815B  +  0 

.87187R2  +  0. 

6C825P3  J 

-0.29190B4 

>  Lt;  • 

FY’80 

FY'8 1 

1 

» 

FORECAST  ACTUAL  PE 

FORECAST  ACTUAL 

PE  ■ 

1st  Qtr 

31,014  27,455  15.88 

35,335 

29,455 

19.96  j 

2nd  gtr 

34,899  35,034  .004 

35,779 

34,938 

■ 

2.41 

• 

3rd  Qtr 

43,464  36,864  17.90 

44,712 

37,182 

• 

20.25 

4th  gtr 

43,116  36,562  17.92 

45,039 

38,802 

16.07  i 

TOTAL 

153,293  135,915 

160,865 

140,377 

jj 

* 

FY'82 

» 

FORECAST 

ACTUAL 

PE 

* 

1st  otr  37,453 

30,330 

23.48 

• 

* 

2nd  Qtr  36,308 

35,040 

1.02 

: 

»■ 

3rd  Qtr  41,463 

41,282 

10.13 

l 

4th  Qtr  46,196 

45,739 

1.00 

TOTAL  165,420 

153,194 
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COMPARISON  OF  CEP  ESTIMATORS  FOR 
ELUPriCAL  NORMAL  ERRORS 
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Naval  Surface  Weapons  Center 
Dahlgren,  Virginia 


Introduction 


-C 


A  ccmmon  parameter  for  describing  the  accuracy  of  a  weapon  is  the/  circular 
probable  error ^  generally  referred  to  as  CEP.  CEP  is  simply  the  bivariate  analog 
of  the  probable  error  of  a  single  variable  and  measures  the  radius  of  a  mean 
centered  circle  which  includes  501  of  the  bivariate  probability  mass.  In  the  case 
of  circular  normal  errors  where  the  error  variances  are  the  same  in  both  directions, 
CEP  can  be  expressed  in  terms  of  the  common  standard  deviation,  and  estimators  are 
easily  formulated  and  compared.  In  the  case  of  elliptical  normal  errors,  CEP  can¬ 
not  be  expressed  in  closed  form,  and  hence,  estimators  are  less  easily  formulated. 
The  problem  addressed  herein  is  the  comparison  of  CEP  estimators  for  the  elliptical 
case  based  on  some  of  the  commonly  used  CEP  approximations.  . — _ _ ^ 

It  will  be  instructive  to  first  review  the  case  of  circular  normal  errors. 

In  general,  it  will  be  assuned  that  the  errors  in  the  X  and  Y  directions  are 

2  2 

independent  with  mean  zero  and  variances  a  and  a  ,  respectively.  Under  the 

X  y 

2  2  2 

circular  normal  assumption,  ■  a  ■  a  and  the  bivariate  distribution  of 
errors  is  given  by 


f_(x,y)  - 


-(x2+y2)/2o2 
c  # 


x,y  < 


The  distribution  of  R  -  (X  +  Y  )  is  easily  derived  and  found  to  be 


gc(r) 


r 

7 


,-rW 


r  >  0 


C2 


This  is  the  well  known  Rayleigh  distribution  with  cumulative  distribution 
function 


2  2 

P(R  <  r)  -  GcCr)  -  1  -  e'r  /2a  .  (3 

By  definition,  Gc(CEP)  ■  .5,  and  the  solution  of  equation  (3)  yields  the  well- 
known  expression 

CEP  ■  C-24n  (.SO)]*1  o  -  1.1774a  .  (4 

Four  estimators  for  CEP  in  the  circular  case  were  formulated  and  compared  by 
Moranda  (1959) . 

Consider  now  the  case  of  elliptical  normal  errors.  Here  the  variances  are 
unequal,  and  the  bivariate  distribution  of  errors  is  given  by 


*p(x,y) 


1 

TrrcT.o 


* 


[(x/a 


)2. 


x  y 


Cy/ay)  ‘ 


<  x,y 


<  OO 


(5: 


For  this  case,  the  distribution  of  the  radial  error  R  was  derived  by  Chew  and 
Boyce  (196.1)  and  has  form 


and  I  is  the  modified  Bessel  function  of  the  first  kind  and  zero  order,  i.e. , 
o 

TT 

I  (x)  -  I  1  e'x  cos9ae . 

O  7T  0 

The  cumulative  distribution  function  for  R  is  denoted  by 

r 

P(R  <  r)  -  GB(r)  -  5  gfi(t)dt,  (7) 

but  it  cannot  be  expressed  in  closed  form.  Hence,  the  radius  of  the  50%  circle 
for  the  elliptical  case  cannot  be  expressed  by  a  simple  formula  as  it  was  in  the 
circular  case.  One  has  to  solve  Gg(CEP)  ■  .5  by  numerical  methods  or  by  referring 
to  tables  prepared  by  Harter  (I960),  DiDonato  and  Jamagin  (1962),  and  others.  To 
avoid  using  these  tables  or  numerical  procedures  for  CEP  evaluation,  a  number  of 
approximations  have  been  developed  over  the  years.  Five  of  these  approximations 
have  been  chosen  for  examination.  They  are  designated  below  as  CEP-^  through  CEP5: 
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(11) 


CEP,  ■  .565  omav  +  .612  o._ ,  cr  ,_/o  >.  .25 

4  max  min’  min'  max 


.667  a  +  .206  a.,  a./o  <  .25 
max  min’  mm  max 


CEP^  and  CEP2  were  taken  from  Groves  (1961),  CEP^  was  established  by  Grubbs  (1964), 
CEP^  is  a  piece-wise  linear  combination  of  the  standard  deviations,  and  CEP5  was 
also  established  by  Grubbs  (1964)  using  a  Wilson-Hilferty  transformation  of  the 
chi-square  in  CEPj.  Plots  of  each  approximation  versus  the  true  CEP  as  a  function 
^in^max  are  s^ov,n  Fl8ures  1  through  5.  These  give  a  fairly  good  indication 
of  how  well  each  performs.  It  is  seen  that  CEP1  deteriorates  rapidly  as  we  depart 
from  the  circular  case  (for  which  CBP^  degenerates  to  1.1774a),  CEPZ  is  reasonably 
good  if  the  ratio  ainin/CTmax  *s  not  less  about  .2,  CEP^  appears  good  for  all 
ratios,  and  CBP^  and  CBPg  appear  good  to  a  lesser  extent  for  all  ratios. 

If  these  approximations  were  used  only  as  approximations  for  assumed  values 
of  the  error  variances  (as  one  does  in  wargaming  and  round  requirement  studies) , 
then  there  would  be  no  estimation  problem.  However,  in  many  cases,  weapons 
analysts  are  using  estimates  of  the  variances  in  these  approximations  (based  on 
sanple  data)  to  form  estimates  of  CEP.  Hence,  the  problem  now  becomes  an  estima¬ 
tion  problem  instead  of  an  approximation  problem.  In  particular,  the  problem 
addressed  in  this  paper  is  that  of  comparing  the  five  estimators  for  CEP  formed 
by  replacing  the  population  variances  in  equations  (8)  through  (12)  with  sample 
variances  Sx  ■  I  X^/n  and  ■  E  Y^/n.  (In  these  expressions,  and  are  the 
recorded  errors  in  the  X  and  Y  directions,  respectively,  for  the  ith  impact  and 
n  is  the  number  of  sample  impacts.)  These  estimators  will  be  referred  to  as 

A  A 

CEP^  through  CEP5  in  the  discussion  '.vhich  follows. 


312 


Methodology 


Measures  of  comparison  employed  in  this  study  were  the  mean  squared  error 
(NEE) ,  expected  confidence  interval  length,  and  confidence  interval  confidence. 

A 

With  regard  to  the  former,  the  MSE  of  an  estimator  9  for  a  parameter  0  is  defined 
in  the  usual  sense,  i.e., 


MSE (6)  ■  E(6  -  e)2  -  v(e)  +  b2  (e) 


where  E  represents  expectation,  V  represents  variance  and  B  represents  bias.  It 
was  chosen  because  it  accounts  for  bias  as  well  as  variance  and  all  five  estimators 
are  biased  for  CEP  except  in  the  degenerate  circular  case.  With  regard  to  the 
second  measure,  it  was  chosen  because  it  too  accounts  for  bias  as  well  as  variance 
but  in  the  sense  of  interval  estimation  vice  point.  These  computations  were  based 
on  approximate  distributions  of  CEP  estimators  and  did  not  provide  intervals  with 
specified  confidence  in  all  cases.  Hence,  the  third  measure  was  included  to 
estimate  the  true  confidence. 


The  computation  of  these  measures  was  straightforward  but  not  simple  due  to 
the  complexity  of  the  estimators.  Recall  that  1,  3  and  5  each  involve  radicals 
of  linear  combinations  of  sample  variances  and  estimators  2  and  4  involve  linear 
combinations  of  sample  standard  deviations.  Hence,  the  sampling  distributions 
were  approximated.  The  approximations  were  achieved  by  matching  the  variance  of 
estimators  3  and  5  with  the  variance  of  the  chi-square  distribution  and  by  match¬ 
ing  the  variance  of  estimators  2  and  4  with  the  variance  of  the  chi  distribution. 
Estimator  1  was  simply  approximated  by  a  chi-square  with  2n  degrees  of  freedom. 
This  distribution  is  exact  only  in  the  circular  case  and  was  included  to  show  how 
poorly  it  becomes  when  eccentricity  of  the  distribution  increases.  The  approxi¬ 
mations  are  shown  in  Figure  6  and  are  discussed  in  more  detail  in  the  next 
paragraph. 
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Figure  6  provides  a  s urinary  of  approximate  distributions  for  each  CEP. 

1  , 

and  defines  several,  multiplicative  factors,  1C  ,  eccentricity  c  and  degrees  of 
freedom  v  and  v* .  v*  does  not  have  a  simple  form  and  is  described  below. 

Because  estimators  3  and  5  are  of  the  same  general  form,  the  distribution  j 

of  the  squares  of  both  was  approximated  by  matching  the  variance  of 


tsx  *  sv> 

1^9 


(13) 


with  2  v' , 
that  v'  - 


the  variance  of  a  chi-square  with  v'  degrees  of  freedom.  It  was  found 
2  2 

nv  where  v  ■  -  .  Expression  (13)  can  be  rewritten  as 

c  +  1 


v'  cep: 


o2  X 

y 


iM 


where  i  ■  3  or  S 


(14) 


to  conform  to  the  expressions  in  Figure  6. 

Estimators  2  and  4,  representing  linear  combinations  of  the  standard 
deviations,  were  approximated  by  matching  the  variance  of  a  chi  with  v*  degrees 
of  freedom  with  the  variance  of 


(v*)*1  (Sx  ♦  Sy) 
ax  +  ay 


The  variance  of  expression  (15)  was  found  to  be 


v*  (1  -  HZ(n))  1+c 


(l+c) 


(15) 


(16) 


and  the  variance  of  a  chi  with  v*  degrees  of  freedom  is 


where  H(x) 


Upon  equating  the  two,  we  find  that  v*  satisfies 


H(v*) 


(1  -  H2(n))  -iiS-y 

(l+c)Z 


H 


and  can  be  obtained  from  a  table  of  inverse  solutions  of  the  H  function. 

The  approximate  distributions  allow  one  to  derive  approximate  mean  squared 

A 

errors  for  the  CEPi  estimators  which  are  given  in  Figure  7.  The  JL  coefficients 

A 

are  defined  as  before  and  Bias(CEP^)  is  defined  as 

E(CEP^)  -  True  CEPi  for  i  -  1,  2,  . ..,  5. 


In  general,  a  is  a  scale  factor  representing  the  maximum  o  value;  however,  in 

^  A  A 

the  examples  given  here  o  is  always  equal  to  1.  Note  that  MSE(CEP2)  and  MSEfCEP^) 
can  be  expressed  in  exact  rather  than  approximate  form. 

Since  a  point  estimate  may  not  provide  adequate  information,  approximate 
95%  confidence  intervals  were  constructed  for  each  estimator  using  the  distributions 
discussed  above.  The  approximate  100(l-a)%  confidence  limits  for  CEP  are  given 


by 


CEP. 

CEPi 

_(xv1,l-«/2/vi)  H  ( 

xvi,o/2/vi)  ^ 

A 

where  CEP^  is  the  ith  estimator  and  equals  the  degrees  of  freedom  associated 

/s 

with  CEP^.  Expected  confidence  interval  widths  can  then  be  computed  and  used  as 
measures  of  comparison  bet\  e  i  stimators.  Clearly,  if  one  could  compute  exact 
95%  confidence  intervals,  comparison  of  interval  widths  would  be  straightforward. 
However,  only  approximate  intervals  can  be  obtained  and  the  confidence  associated 
with  each  interval  must  be  computed  before  a  complete  evaluation  can  be  made. 
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Confidence  was  estimated  using  10,000  Monte  Carlo  replicates  for  samples 
of  size  5,  10  and  20  and  measuring  the  percentage  of  time  the  true  CEP  fell 
within  the  interval.  Confidence  and  expected  confidence  interval  widths  were 
then  jointly  examined. 

Results 

Ihe  object  of  this  study  was  to  examine  and  evaluate  the  behavior  of 
several  candidate  CEP  estimators  over  a  wide  range  of  conditions.  Sample  sizes 
ranged  from  5  to  400  and  eccentricities  ranged  from  c  ■  1,  the  circular  case  to 
c  ■  20,  a  highly  elliptical  case.  Extreme  values  of  the  sample  size  and 
eccentricity  may  be  infrequently  encountered  but  were  included  for  completeness. 
Clearly,  an  estimator  behaving  poorly  under  circumstances  unlikely  to  be  observed 
should  not  be  disregarded  as  a  viable  candidate. 

Prior  to  determining  approximate  distributions  and  mean  squared  error 
(MSE)  approximations  for  the  estimators,  a  Monte  Carlo  simulation  was  developed 
for  computing  the  variance,  bias,  average  squared  error  CASE)  and  standard  error 
for  each  estimator  at  each  of  three  sample  sizes  (n  ■  5,  10,  20).  The  simulated 
ASE's  were  used  as  a  check  against  MSE  approximations  which  were  subsequently 
computed. 

Upon  comparing  the  simulated  ASE's  against  results  of  the  MSE  approximations 
for  sample  sizes  5,  10,  and  20,  it  became  evident  that  MSE  approximations  were 
inadequate  for  estimators  3  and  5.  In  fact,  in  the  mid-range  of  the  eccentricity, 
c,  the  MSE  for  3  and  5  differed  from  the  simulated  values  of  ASE  by  as  much  as 
three  times  the  standard  error.  For  this  reason,  the  simulated  ASB  values  are 
presented  in  Figure  8  while  the  approximate  MSE  values,  found  suitable  for 
larger  samule  sizes,  are  shown  in  Figure  9. 

Despite  some  fiuctuation  at  c  -  .05,  Figures  8  and  9  show  estimators  2 
through  5  producing  fairly  close  results.  As  the  sanple  size  increased, 
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estimator  4  exhibited  the  smallest  mean  squared  error  and  appeared  to  be  the 
most  satisfactory  point  estimator. 

Figures  10  and  11  contain  expected  confidence  interval  width  and  confidence 
interval  confidence,  respectively.  If  the  computation  of  exact  95%  confidence 
intervals  were  possible,  a  straightforward  selection  of  the  estimator  producing 
the  narrowest  width  could  be  made.  However,  the  approximate  confidence  intervals 
have  varying  levels  of  confidence  associated  with  them,  all  of  which  under¬ 
estimate  or  overestimate  the  desired  95%  level.  It  appears  that  the  wider  lengths 
are  associated  with  higher  confidence  and  the  narrower  widths  with  the  lower 
confidences  so  that  a  true  comparison  is  not  really  possible.  However,  it  is 
evident  that  estimators  2  through  5  do  not  distinguish  themselves  as  being  far 
superior  or  grossly  inferior  to  one  another.  This  is  essentially  the  same  result 
obtained  from  the  MSB  comparisons. 

In  simmary,  unless  c  is  very  small,  estimators  2  through  5  produce 
reasonably  close  results.  If  confidence  intervals  are  not  desired,  estimator  4 
would  be  an  acceptable  choice.  Otherwise,  estimator  3  is  recommended  due  to 
ease  of  confidence  interval  computability. 
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CONFIDENCE  INTERVAL  LENGTHS 
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ABSTRACT.  In  the  boundary  layer,  wind  fluctuations  during  short  time 
Intervals  in  the  magnitude  of  seconds  are  commonly  interpreted  in  terms  of 
turbulence  theory.  According  to  Kolmogorov,  Obukhov  and  Corrsin,  this 
requires  a  -5/3  slope  in  a  diagram  of  the  power  spectrum  versus  the  wave 
length  in  double  logarithmic  coordinates.  The  turbulence  characteristics 
fade  with  averaging  time  of  the  wind  measurements. 

Investigations  of  one-  ana  six-second  tower  measurements  at  Redstone 
Arsenal  ar.d  Otis  Air  Force  Base  revealed,  however,  that  these  small  scale 
fluctuations  more  frequently  show  associations  with  white  or  red  noise  rather 
than  turbulerce  characteristics  In  the  spectrum.  It  will  be  Illustrated 
that  the  turbulence  slope  is  a  special  case  of  red  noise,  c _ 

I.  INTRODUCTION.  In  the  past  two  decades  wind  sensors  with  high  sens¬ 
itivity  and~THort—tTme  responses  were  developed.  These  instruments  made  it 
possible  to  measure  wind  fluctuations  on  *he  scale  of  seconds  or  even  for 
shorter  time  intervale.  In  the  boundary  layer  these  short  time  fluctuations 
are  mostly  analyzed  or  interpreted  in  terms  of  frictional  or  convective 
turbulence  theo  /.  It  is  sometimes  overlooked,  howevar,  that  these  fluctu¬ 
ations  may  be  random  (whitei  noise  or  simply  include  persistence  which 
produces  a  red  noise  spectrum. 

In  the  subsequent  sections  a  detailed  analysis  of  power  spectra  of  che 
wind  is  performed  for  observations  from  one-  and  six-second  tower  measure¬ 
ments  at  Redstone  Arsenal,  Alabama,  and  Otis  ArB,  Mass.,  by  Gil’  u-v-w 
anemometers  (see  Gill,  1975).  It  is  delineated  that  the  slope  of  some 
spectra  (in  double  logarithmic  coordinates)  differs  from  tse  expected  -5/3 
law  of  turbulence  (inertial  subrange),  but  it  follows  a  pattern  of  persistence. 

The  slope  of  power  spectra  for  red  noise  was  studied.  It  can  be  shown 
that  a  fi^st  lag  correlation  of  0.85  and  exponential  red  noise  produce  a 
slope  of  -5/3  in  the  spectra  for  medium  and  short  waves,  although  for  the 
empirical  data  in  this  investigation  the  first  lag  correlation  producing  a 
-5/3  slope  was  slightly  higher  (see  Figure  3).  Red  noise  in  the  data  series 
can  produce  the  same  slope  as  turbulence,  although  turbulence  and  red  noise 
spectra  are  not  identical  and  differ  in  the  region  of  long  waves  and  at  the 
very  tail  end  of  short  wave  length. 

Finally,  the  relationship  in  time  averaging  of  wind  components  for  data 
from  Redstone  and  Otis  is  analyzed.  As  expected,  the  variance  of  the  data 
decreases  with  increasing  average  time.  The  first  lag  correl ations  for  the 
averaged  data  sets,  however,  do  not  display  a  uniform  pattern. 
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II.  THE  POWER  SPECTRUM  OF  TURBULENCE.  It  is  customary  to  present  the 
time  fluctuation  of  the  U-component  of  the  wind  (in  direction  of  the  mean 
flow)  by: 

U  =  us  +  Ut  (1) 

where  U$  is  the  "stationary"  part  and  Ut  the  "turbulent  fluctuation."  The 
other  wind  components,  V  and  W,  follow  in  analogy. 

The  stationary  part  is  usually  the  mean  value,  U$  =  U.  If  U  changes  in 

time,  the  series  must  be  considered  as  nonstationary  and  U$  may  represent 

some  changing  quantity,  although  the  change  may  be  slow.  Nonstationary  time 
series  may  show  complex  patterns,  although  the  change  in  U$  would  not  affect 

ihe  part  o"  a  power  spectrum  towards  waves  of  short  duration.  Techniques  to 
separate  Uc  from  Ut  have  been  developed  (e.g.,  Essenwanger  and  Billions,  1965). 

For  this  investigation,  U$  is  considered  to  be  constant  over  a  homogeneous 

time  period. 

The  turbulent  fluctuation  requires  (by  definition  of  Us  =  U) 

Z  Ut/N  =  Ut  =  0  (2) 

but  the  variance: 

E  U2/N  -  aj  *  0  (3) 

t 

In  addition,  for  turbulence  In  the  inertial  subrange,  the  energy  spectrum  as 
a  function  of  the  (standardized)  wave  number  K  follows: 

E(K)  -  ae2A  K-s/3 

according  to  Kolmogorov-Obukhov-Corrsln  (e.g.,  Tennekes  and  Lumley,  1973,  p. 

266;  Priestly,  1959,  p.  61;  Hinze,  1959,  p.  194,  etc.).  This  leads  to  a 
squared  standardized  amplitude  L1  of  a  Fourier  series  for  the  time  series  of 
turbulence  data:  J 

In  E(K)  =  In  Lj  =  const  -(5/3)  In  K.  (4) 

(For  Lj  of  the  power  spectrum,  see  Tukey,  1949;  Blackman  and  Tukey,  1958;  or 
Cooley  and  Tukey,  1965.)  Thus,  Lj  will  show  a  slope  of  -5/3  in  double  loga¬ 
rithmic  coordinates  with  Lj  as  the  ordinate  and  In  K  as  the  abscissa. 

Equation  (4)  is  referred  to  In  this  study  whenever  the  slope  angle  is  discussed. 

Wind  observations  recorded  at  one-second  time  Intervals  with  the  u-v-w 
Gill  anemometers  (see  Gill,  1975)  were  made  in  1974  on  towers  at  Redstone 
Arsenal,  Alabama,  and  the  power  spectrum  of  the  AU  component  for  a  combination 
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of  levels  was  established.  Figure  1  provides  an  example  for  the  power  spectrum 
All  of  the  windshear  from  18-  to  30-foot  heights,  where  AU  is  the  difference  of 
the  U-components  from  two  respective  heights.  The  slope  of  the  power  spectrum 
(obtained  by  regression  analysis)  shows  -1.63,  which  is  close  to  -5/3.  The 
measurements  over  a  one-hour  time  period  fulfilled  the  definition  of  stationary 
data. 


Later  recordings,  at  six-second  time  intervals,  were  obtained  (1981)  from 
Otis  AFB,  but  the  power  spectra  of  the  AU  windshears  did  not  delineate  a  -5/3 
slope  for  this  set  of  data.  A  detailed  analysis  of  the  U-components  themselves 
revealed  that  the  wind  components  at  the  individual  levels  do  not  follow  a  power 
spectrum  of  -5/3.  Consequently,  it  may  not  be  expected  that  the  power  spectra 
of  the  windshear  show  a  -5/3  slope.  An  example  for  the  power  spectrum  (double 
logarithmic  coordinates)  of  the  U-component  at  an  altitude  of  10  feet  for  the 
tower  data  set  under  Investigation  Is  illustrated  in  Figure  2. 

At  first  it  was  considered  that  the  difference  in  the  slope  angle  could 
be  attributed  to  the  sampling  rate.  Some  change  in  the  angle  may  be  expected 
for  an  assumption  of  red  noise  (see  Section  III),  but  turbulence  In  the  Inertial 
subrange  would  require  that  the  slope  should  still  be  around  -5/3. 

The  power  spectrum  (Figure  2)  Is  truncated  at  waves  of  12-second  duration 
compared  with  Figure  1,  which  extends  to  two  seconds,  but  the  slope  should  not 
have  changed  that  significantly.  Corresponding  with  Figure  1,  spectral  values 
from  wave  length  corresponding  to  30  seconds  to  12  seconds  do  not  produce  a 
-5/3  slope  In  Figure  2. 

One  may  suggest  that  the  25  November  1981  data  stem  from  a  nonstationary 
time  series.  Indeed,  there  Is  a  break  In  U  at  about  the  middle  of  the  time 
series.  However,  as  pointed  out  previously,  fluctuation  of  would  affect 

the  amplitudes  of  waves  of  longer  duration  and  should  not  alter  the  tail  end 
of  the  spectrum  for  waves  with  short  duration.  Thus,  nonstationary  behavior 
is  not  reflected  in  cnanges  of  the  wave  spectrum  of  shorter  length.  A  split 
of  the  data  Into  the  two  different  periods  provided  the  same  pattern  as  Illu¬ 
strated  in  Figure  2. 

A  further  possibility  for  the  lower  slope  angle  in  Figure  2  data  Is  the 
assumption  that  the  data  do  not  show  turbulence  in  the  inertial  subrange,  or 
the  fluctuations  are  governed  by  different  laws.  As  an  alternative  hypothesis, 
a  white  or  red  noise  pattern  will  be  studied  next. 

III.  WHITE  AND  RED  NOISE.  It  is  common  knowledge  that  meteorological 
time  series  show  persistence.  Persistence  in  signal  processing  is  usually 
called  red  noise,  while  white  noise  indicates  Independence  in  the  data  sample. 
White  noise  leads  to  a  power  spectrum  where  al1  (squared)  amplitudes  are  equal 
except  for  random  fluctuations.  Red  noise  can  be  expressed  by  several  models, 
although  two  primary  forms  have  been  customary  in  geoscience  (e.g.,  Taubenhelm, 
1969). 

In  various  texts  the  autocorrelogram  in  turbulence  analysis  is  assumed 
to  be  Gaussian  noise  (e.g.,  Tennekes  arid  Lumley,  1973;  Hinze,  1959,  etc.). 
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This  form  was  ruled  out  by  the  author  for  the  investigated  data  because  of 
dlsslml lari ty  in  the  correlogram  and  spectrum,  and  the  use  of  the  exponential 
red  noise  model  by  other  authors  such  as  Pasquill  (1962),  Fichtl  and  McVehil 
(1970),  or  Hanna  (1979).  (See  also  Stewart,  1981.) 

The  form  chosen  here  is  the  exponential  red  noise  which  is  identical  with 
a  first  order  Markov  chain.  In  terms  of  an  autocorrelogram  we  can  write: 

Pk  =  P*  (5) 

where  pc  Is  the  first  lag  correlation  and  k  =  0 . m.  This  series  Is 

identical  with: 

Pt  *  exp  (-bt)  (6) 

where  t>0,  b  =-ln  pc>0,  pc>0  (see  Box  and  Jenkins,  1970,  or  Essenwanger, 

1980).  The  corresponding  power  spectrum  Is: 

Lj  =  (2/b)  /  [1  +  it2 j 2/b]  (7) 

(see  Taubenhelm,  1969,  etc.).  Gilman  et  al .  (1963)  derived: 

Lk  =  [(1  -  p*)  /  (1  +  p|  -  2  pc  cos  km/m]  /  m  (8) 

which  Is  a  good  approximation  for  p  <  0.9. 

The  power  spectrum  of  white  noise  can  be  stated  as: 

Lk  »  const  (9) 

and  Is  produced  from  pQ  «  1 ,  Pk  =  0  for  all  k  >  0. 

Figure  3  represents  the  power  spectrum  of  (exponential)  red  noise  for 
a  variety  of  first  lag  correlations  from  pc  =  0.9  down  to  0.1  for  a  normal¬ 
ized  maximum  lag  of  m  =  60.  The  power  spectra  show  a  variety  of  slope  angles 
ranging  from  -1.78  for  p  =  0.9  to  about  -0.20  for  p  =  0.1.  Thus,  the 

W  I# 

power  spectrum  exhibited  in  Figure  2  could  come  primarily  from  data  assoc¬ 
iated  with  red  noise  of  pc=  0.2.  If  the  red  noise  data  concept  Is  the  generating 

background  of  the  wind  fluctuation,  then  the  slope  In  the  power  spectrum 
would  change  with  a  change  In  the  sampling  time  Interval. 

Figure  4  provides  an  example  of  the  autocorrelogram  of  data  taken  at 
Redstone  (19  May  1974)  at  one-second  sampling  time  and  the  corresponding 
series  of  the  same  data  at  six-second  time  intervals.  In  the  latter,  every 
sixth  correlation  coefficient  appears.  While  the  first  lag  correlation  for 
one-seconds  Is  pi=  0.97,  the  value  drops  to  Pi=  0.80  for  six  seconds.  A 

corresponding  change  In  the  slope  of  the  power  spectrum  Is  observed.  Under 
the  assumption  of  exponential  red  noise,  the  change  in  angles  Is  very  small. 

It  amounts  to  about  four  degrees.  (From  pc  =  0.97  to  pc  =  0.8,  the  slope 

reduces  from  -1.85  to  -1.56,  or  in  angles  from  61.6°  to  57.3°.) 
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CIO.  4 


AUTOCORRELATION,  U-COMP. 


Figure  4  supports  the  conjecture  that  exponential  red  noise  is  a  primary 
factor  In  the  1974  data  series.  Only  correlation  coefficients  pk  for  k  >  50 

deviate  from  the  pattern.  These  coefficients  k  >  50  are  not  significantly 
different  from  zero,  however,  although  they  appear  dissimilar  to  the  exponential 
red  r.jise  line. 


Figure  5  exhibits  the  empirical  relationship  between  the  slope  angle  and 
the  first  lag  correlation  p  of  the  U-component  and  All  of  observed  wind  data. 

v# 

These  37  samples  were  prepared  from  power  spectra  of  the  19  August  1974  (Red¬ 
stone)  and  25  November  1981  (Otis  AFB)  data.  The  regression  line  deviates  only 
slightly  from  the  expected  line  of  pure  red  noise  (Figure  3).  Six  samples 
(dots)  of  the  37  samples  represent  the  slopes  of  time  difference  windshears 
(data  19  August  1974): 

AtS  •  [(AtU)J  +  (AtV)*  ]*  (1C) 

where  At  denotes  the  time  difference.  In  order  to  create  Independent  data 

sets,  the  original  data  series  was  reduced  by  accepting  only  every  t-th  value. 

The  autocorrelogram  for  ■  8  seconds  showed  a  p;  •  0.65,  which  slowly  decreased 

to  p60  *>  0.48.  Thus,  the  very  slow  decline  produces  the  outlier  (pc  =  0.65, 

slope  angle  -5°)  and  can  be  explained  as  a  truncated  but  non-exponential  auto¬ 
correlogram.  However,  the  physical  Interpretation  of  the  high  persistence  In 
this  particular  data  series  needs  further  Investigation. 


The  relationship  In  Figure  5  and  a  comparison  with  Figure  3  render  the 
conclusion  that  p  Is  about  0.85  If  the  slope  should  agree  with  the  required 

slope  of  turbulence  In  the  Inertial  subrange.  Consequently,  a  slope  angle  of 
-5/3  can  also  be  produced  by  a  data  series  whose  persistence  factor  agrees 
with  an  exponential  red  noise  model  of  p  between  0.8  to  0.9. 

w 


In  turbulence  analysis  an  Integral  scale  Is  defined: 
p(t)dt 


(U) 


For  time  series  records  at  a  given  point  is  the  Eulerlan  Integral  scale 

(e.g.,  Tennekes  and  Lum'ley,  1972,  p.  275,  etc.).  It  can  be  related  to  the 
Eulerlan  length  scale  of  turbulence: 


Le.U*s  (12) 

(e.g.,  Ivanov  and  Kllnov,  1962,  Tennekes  and  Lumley,  1972,  Weber  et  al . ,  1982, 
etr. ) . 


In  the  exponential  model  IT  Is  a  unique  function  of  p  ,  the  first  lag 

9  w 

correlation,  and  pc  Is  related  to  the  slope.  If  turbulence  were  to  follow 

the  exponential  red  noise  model,  it  implies  that  turbulence  in  the  Inertial 
subrange  requires  a  special  value  of  R-  and  the  length  scale  of  turbulence 

would  change  only  with  a  change  of  U.  If  U  is  constant,  however,  the  slope 
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of  the  power  spectrum  Is  a  function  of  R  alone,  which  Implies  that  it  is  a 

function  of  pc,  and  may  differ  from  -5/3.  Thus  the  power  spectrum  may  show 

other  turbulence  than  the  inertial  subrange,  a  variation  of  the  slope  according 
to  the  type  of  stability  conditions  (e.g.,  Tennekes  and  Lumley,  1972,  Weber 
et  al.,  1982,  etc.),  or  no  turbulence  at  all.  In  the  latter  case,  the  fluctu¬ 
ations  of  the  wind  velocity  follow  a  random  pattern  with  persistence  ^red 
noise). 

The  associations  illustrated  in  Figures  3,  4,  and  5  leave  another  Inter¬ 
pretation  open.  Let  us  return  to  equation  (5)  and  write: 

PM  *  °c  a  e  (13> 

Using  M  as  a  scaling  factor,  the  exponential  autocorrelogram  can  be  standard¬ 
ized,  e.g.,  assume  that  e  =  0.001.  In  a  first  case  of  pc  *  0.9,  we  find  Mi  = 

65.5.  Under  an  assumption  of  p  =  0.5,  we  derive  M*  =  10.0.  Consequently, 

V 

Mi  =  6.5  Mi  .  Let  us  assume  that  we  have  sampled  our  records  every  six  seconds 
and  have  found  a  first  lag  correlation  of  pc  =  0.5.  The  autocorrelogram  for 

(roughly)  one-second  sampling  would  Increase  p  to  0.9  and  provide  a  slope  of 

-5/3.  Since  we  have  not  altered  the  data  set,  IJ  would  not  have  changed.  In 
this  case,  the  Lulerlan  length  scale  of  turbulence  L^.  would  have  changed  be¬ 
cause  for  p  *  0.5  Is  smaller  than  IT  for  0.9.  It  must  be  taken  Into 

5  C  S 

account,  however,  that  At  Is  different  In  both  cases  (equation  11).  Otherwise, 
the  slope  angle  changes  as  an  effect  of  the  sampling  rate. 

The  last  example  Illustrates  that  It  is  not  sufficient  to  study  only  the 
slope  of  the  power  spectrum  In  turbulence  analysis.  Other  turbulence  charact¬ 
eristics  must  be  added  In  order  to  come  to  valid  conclusions. 

IV.  INSTANTANEOUS  AND  STANDARD  INSTRUMENTATION.  Redstone  wind  data  taken 
at  one-second  sampling  Intervals  by  the  u-v-w  Si'll  anemometer  and  at  Otis  AFB 
can  be  considered  as  instantaneous  observations,  although  at  Otis  AFB  sampling 
occurred  every  sixth  second.  There  is  a  trend  toward  development  of  even  more 
sensitive  Instrumentation  In  the  future,  using  electro-optical  wind  measuring 
devices. 

The  difficulties  In  the  Interpretation  of  the  slope  angle  which  were 
delineated  In  the  previous  section  are  not  limited  to  Instantaneous  measure¬ 
ments.  Standard  Instrumentation  measurements  resemble  the  application  of  an 
averaging  process  to  Instantaneous  values.  It  may  be  appropriate  to  examine 
briefly  the  effects. 

In  the  averaging  process,  waves  of  short  length  are  truncated  compared 
with  the  original  spectrum.  This  i:>  not  at:  adverse  effect  by  itself.  Under 
the  assumption  of  exponential  red  noise,  the  first  lag  correlation  would 
determine  the  slope. 
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-u  "V  U  V 


1 

0.98 

0.98 

1.28 

0.50 

4 

0.95 

0.92 

1.25 

0.48 

8 

0.90 

0.85 

1.21 

0.46 

16 

0.81 

0.74 

1.15 

0.43 

24 

0.76 

0.65 

1.07 

0.40 

32 

0.68 

0.61 

1.04 

0.38 

48 

0.61 

0.55 

0.92 

0.33 

64 

0.57 

0.51 

0.86 

0.31 

MEAN  tT  *  -  0.60 

F  =>  0.07  M/SEC 

TABLE  2 

EFFECT  OF  AVERAGING 
(Tower  P) 

DATA:  25  NOVEMBER  1981 
(6  SEC  SAMPLING,  LEVEL  10  FT) 


AVERAGE 

PU 

FIRST 

PV 

Q 

[ 

i 

°V 

PU 

SECOND 

PV 

'» 

1 

0.57 

0.40 

2.85 

1.66 

0.44 

0.25 

2.23 

1.34 

5 

0.59 

0.53 

1.63 

0.74 

0.40 

0.40 

1.04 

0.46 

10 

0.61 

0.57 

1.30 

0.55 

0.37 

0.44 

0.70 

0.31 

15 

0.60 

0.59 

1.16 

0.49 

0.39 

0.35 

0.54 

0.27 

20 

0.60 

0.65 

1.05 

0.43 

0.28 

0.39 

0.50 

0.22 

30 

0.60 

0.69 

0.90 

0,38 

MEAN  U  = 

6.08 

V  = 

=  0.0 

M/SEC 

U  =  6.41 

V  = 

0.0  M/SEC 
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It  Is  not,  trivial  to  predict  the  behavior  of  the  first  lag  correlation 
as  a  function  of  the  averaging  Interval.  The  first  lag  correlation  for 
averages  depends  on  the  structure  In  the  covariance  matrix  of  the  data  series 
and  the  decrease  of  the  variance  by  smoothing.  Table  1  provides  an  example 
of  a  declining  first  lag  correlation  with  increase  of  the  averaging  Interval, 
but  the  decline  Is  not  always  found.  Table  2  exhibits  a  complex  pattern.  In 
three  columns  of  Table  2,  the  first  lag  correlation  Increases  with  Increasing 
averaging  time,  while  one  column  shows  a  decrease.  Since  decrease  and  increase 
do  not  simply  relate  to  the  magnitude  of  the  first  lag  correlation  of  the 
original  daca  series,  no  simple  prediction  model  is  applicable. 

Tables  1  and  2  confirm  that  the  variance  decreases  with  increasing  aver¬ 
aging  Interval  in  accordance  with  the  central  limit  theorem. 

The  slope  In  the  power  spectrum  for  the  averaged  data  series  changes  in 
accord  with  the  magnitude  of  p£  (see  also  Figure  5).  In  the  exponential  red 

noise  model,  depends  on  pc,  and  the  previously  discussed  problems  apply. 

It  Is  possible,  however,  that  averaged  values  represent  fluctuations  of  the 
wind  following  a  random  process  with  persistence.  In  this  case,  the  slope 
angle  need  not  be  -5/3.  More  Investigations  are  necessary,  however,  before 
final  conclusions  can  be  made. 

V.  CONCLUSIONS.  It  is  customary  to  consider  the  fluctuation  of  the  wind 
In  the  boundary  layer  during  short  time  Intervals  (e.g.,  seconds)  to  be  In 
agreement  with  turbulence  In  the  inertial  subrange.  While  this  association 
was  confirmed  for  tower  data  at  Redstone  Arsenal,  other  data  from  Otis  AFB 
delineated  a  different  slope  In  the  power  spectrum  (In  double  logarithmic, 
coordinates).  The  detallod  Investigation  revealed  that  these  small-scale 
fluctuations  are  more  likely  to  be  produced  by  a  random  process  with  persistence 
(exponential  red  noise).  This  concept  would  explain  that  the  angle  In  the 
power  spectrum  Is  not  -5/3,  but  Is  a  function  of  the  first  lag  correlation 
(Figure  5). 

A  second  Interpretation  Is  possible.  The  relationship  between  slope  angle 
and  first  lag  correlation  Is  based  on  an  exponential  red  noise  model.  The 
structure  of  turbulence  In  the  inertial  subrange  Is  an  exponential  red  noise 
model  with  a  first  lag  correlation  arounc  h.85  for  fluctuations  of  short 
duration. 

It  was  further  shown  that  averaging  of  the  Instantaneous  wind  measure¬ 
ments  may  lead  to  a  change  In  slope  angle,  although  no  simple  relationship 
with  the  exhibited  behavior  of  the  first  lag  correlation  could  be  derived. 

The  averaging  process  would  tie  together  Instantaneous  measurements  and 
observations  on  less  sensitive  Instruments. 
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A  NEED  FOR  A  METHODOLOGY  FOR 
PRIORITIZATION  OP 
MISSION  AREA  DEFICIENCIES 


RICHARD  T.  MARUYAMA 


METHODOLOGY  BRANCH 
STUDIES  AND  ANALYSIS  DIRECTORATE 
DEPUTY  CHIEF  OF  STAFF  FOR  COMBAT  DEVELOPMENTS 
US  ARMY  TRAINING  AND  DOCTRINE  COMMAND 
FT  MONROE,  VIRGINIA 


Abstract  .  The  US  Artsy  Training  and  Doctrine  Command  (TRADOC)  reprints  the 
battlefield  user  in  developing  doctrine,  training,  force  structure,  and 
nateriel  requirement  a  for  the  future.  To  ensure  these  requirements  stem  from 
an  overall  battlefield  concept  and  are  bgggd  on  Bound  analysis,  TRADOC 
Alffiducte  Mission  Area  Analysis  (MAA)  in'the  twelve  areas. out  lined  in  figure  1. 
Each  area  is  assigned  to  a  TRADOC  center  or  school  for  analyses  and  the 
prioritisation  of  resulting  deficiencies. 

Once  all  the  analyses  are  completed  and  each  proponent  has  prioritised  the 
deficiencies  within  the  mission  area,  TRADOC  must  integrate  and  prioritise  the 
twelve  deficiency  lists  into  a  single  ordered  list  of  battlefield 
deficiencies.  This  single  list  will  guide  the  development  of  programs  and  the 
allocation  of  resources  toward  correcting  deficiencies  in  order  of  their 
importance. 

The  difficulty  is  in  developing  a  prioritization  methodology  which  is 
suf'iciently  structured  and  rigorous  to  produce  consistent  results  from  year 
to  year,  while  being  sufficiently  simple  and  well  defined  tc  e  understood  and 
accepted  by  the  decision  maLers  who  will  use  it. 


1.  Introduction.  The  Amy's  Training  end  Doctrine  Commend  (TRADOC)  end 
Neteriel  Development  enJ  Reediness  Commend  (DARCOM)  ere  forging  e  linkege 
which  will  help  comprise  the  materiel  development  cycle  while  enebling  the 
Amy  to  project  requirements  further  into  the  future.  The  Amy's  Long  Renge 
Reseerch  Development  end  Acquisition  (RDA)  plenning  system  end  TRADOC's 
Mission  Ares  Anelysis  (MAA)  process  combine  to  provide  e  roedmsp  of  how  to  get 
to  the  Amy  of  the  future.  They  provide  e  meens  to  consider  the  future 
implicetions  of  current  decisions  end  e  wey  to  couple  these  ections  with  the 
Plenning,  Programing,  Budgeting,  end  Execution  System  for  resource  ellocetion. 

The  current  Long  Renge  RDA  process,  while  still  in  its  infeney,  fecilitetes 
timely  end  systemetic  modernicstion.  It  recognises  thet  modernisetion  must  be 
coordineted  throughout  e  totel  system  thet  includes  meter iel.  trsining, 
personnel,  logistics,  doctrine,  tectics  end  relsted  system  requirements.  It 
understends  thet  these  components  ere  interreleted;  solutions  in  one  eree 
could  well  ceuse  deficiencies  in  enother*  Only  s  comprehensive  epproech  to 
the  totel  system  will  produce  equipment  thet  meshes  with  the  force  structure, 
trsining,  end  doctrine.  To  implement  the  process,  however,  DARCOM  must 
understood  the  needs  of  the  future  bettleficld,  end  thet  is  where  TRADOC  comes 
into  the  picture. 

Mission  Area  Analysis  (MAA)  allows  the  synthesising  of  information  gained 
through  many  individual  studies  end  analyses  into  e  single,  internally 
consistent  framework.  To  facilitate  the  detailed  analyses  of  the  Amy's 
ability  to  execute  its  wartime  missions,  the  overall  battlefield  concept  is 
divided  into  12  mission  areas.  These  mission  areas  serve  as  the  basis  for 
measuring  the  capabilities  of  the  force  programed  in  the  current  Program 
Objectives  Memorandum  (POM)  to  fight  a  successful  battle  against  a  projected 
threat.  Each  mission  area  was  assigned  to  a  TRADOC  center/school  for  analyses 
and  the  prioritisation  of  resulting  deficiencies. 

Figure  1  shows  the  TRADOC  mission  area  structure  and  proponent  for  each  area. 


MI88ION  AREA 
CLOSE  COMBAT  (HEAVY) 
CLOSE  COMBAT  (LIGHT) 
AVIATION 
AIR  DEFENSE 
COMBAT  8UPP0RT 
ENGINEERING,  & 

MINE  WARFARE 
COMBAT  8ERVICE  8UPP0RT 
FIRE  8UPF0RT 
BATTLEFIELD  THEATER 
NUCLEAR  WARFARE 
NUCLEAR.  BIOLOGICAL, 
CHEMICAL 

COMMAND  &  CONTROL 

COMMUNICATIONS 
INTELLIGENCE  & 
ELECTRONIC  WARFARE 


nosmvj 

US  ARMY  ARMOR  CENTER,  FT  KNOX,  KY 
US  ARMY  INFANTRY  CENTER,  FT  BENNING,  GA 
U8  ARMY  AVIATION  CENTER,  FT  RUCKER,  AL 
U8  ARMY  AIR  DEFENSE  CENTER,  FT  BLI88,  TX 
US  ARMY  ENGINEER  CENTER,  FT  BILVOIR,  VA 


US  ARMY  LOGISTICS  CENTER,  FT  LEE,  VA 
US  ARMY  FIELD  ARTILLERY  CENTER,  FT  SILL,  OK 
US  ARMY  COMBINED  ARMS  CENTER,  FT  LEAVENWORTH, 
KS 

US  ARMY  CHEMICAL  SCHOOL,  FT  MCCLELLAN,  AL 

US  ARMY  COMBINED  ARM8  CENTER,  FT  LEAVENWORTH, 
KS 

US  ARMY  SIGNAL  CENTER,  FT  GORDON,  GA 
US  ARMY  INTELLIGENCE  CENTER,  FT  HUACHUCA,  A Z 


Once  the  MAAs  are  complete,  work  begins  to  integrate  the  lists  of  deficiencies 
from  each  mission  area  and  prioritize  them  into  a  single  ordered  list  of 


battlefield  deficiencies.  This  single  list  will  guide  the  development  of 
programs  and  the  allocation  of  resources  toward  correcting  deficiencies  in 
order  of  their  importance. 


Ha  JtotllfltolflKli 

The  TRADOC  prioritisation  process  consists  of  four  phases  (see  Table 

1). 


TABLE  1 

4  PHASE  APPROACH 


I.  ESTABLISH  LIST  OF  MAJOR  DEFICIENCIES  IT 
MISSION  AREAS 

II.  PRIORITIZE  DEFICIENCIES  WITHIN  MISSION 
AREAS 

III.  INTECRATE 12  MISSION  AREA  DEFICIENCY  LISTS 

IV.  A60REGATE  RESULTS  INTO  ONE  PRIORITIZED 
LIST 


a.  Phase  I  -  Establish  lists  of  major  deficiencies  by  mission  area  (see 
figure  1). 

A  atravman  list  of  major  deficiencies  was  developed  for  each  of  the  12 
TRADOC  mission  areas  by  a  BQ  TRADOC  panel  comprised  of  DCSCD,  DC8DOC  and  DOST 
representatives.  Each  strawman  list  was  then  forwarded  to  the  appropriate 
mission  area  proponent  for  review  and  input.  The  mission  area  proponents 
submitted  recommended  corrective  actions  (three  to  five)  for  each  identified 
major  deficiency.  The  number  of  corrective  actions  was  arbitrarily  fixed,  st 
3  to  5  with  the  intent  being  only  to  capture  the  thrust  of  significant 
corrective  actions.  Mission  area  proponents  then  provided  the  integrating 
centers  and  HQ  TRADOC  with  a  copy  of  the  revised  deficiency  list  with 
recommended  corrective  actions  for  final  review.  The  headquarters  review 
included  a  screen  for  consistency  in  describing  deficiencies.  The  result  of 
Phase  I  is  12  separate  lists  of  major  deficiencies.  Figure  2  shows  the 
development  of  the  12  separate  deficiency  lists  for  each  mission  area. 


PHASE I 


FIGURE  2 

b.  Phase  II  -  Prioritise  deficiencies  within  each  mission  area. 

Each  mission  area  proponent  then  prioritised  their  list  of  major 
deficiencies  obtained  in  Phase  I*  This  assessment  was  conducted  considering 
the  Army's  programed  forces  using  systems  scheduled  for  fielding  or  fielded  by 
1987. 


PHASI II 


FIGURE  3 


The  technique  of  pairwise  comparisons  was  used  to  prioritise  the  list  of 
major  deficiencies.  It  involves  asking  mission  area  experts  (approximately 
30)  to  independently  consider  the  list  of  mission  area  deficiencies,  compare 
the  deficiencies  two  at  a  time,  and  sequentially  determine  their  relative 
importance.  Figure  3  shows  the  pairwise  comparison  process  used  to  prioritise 
each  mission  area  deficiency  list. 

A  sample  of  the  survey  form  for  the  pairwise  comparisons  is  shown 
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in  figure  4.  The  ranking  of  a  particular  deficiency  waa  determined  by  the 
number  of  timae  (frequency  count)  it  waa  judged  to  be  moat  important  in 
accordance  with  the  above  criteria.  Individual  judgmenta  were  treated 
equally.  The  deficiency  frequency  counta  from  each  miaaion  area  expert  vaa 
aggregated  and  then  normalised  between  0.0  and  1.0.  This  proceaa  produced  a 
cardinally  ranked  liat  of  deficienciea  (i.e.,  the  order  aa  well  aa  the 
interval  between  each  deficiency  was  eatabliahed).  The  liat  of  cardinally 
ranked  deficienciea  along  with  the  completed  survey  forms  was  then  returned  to 
HQ  TRADOC. 


PAIRWISE  COMPARISON  EXAMPLE 


//A  /* 


a  •  c  •  i  i 


FIGURE  4 


c.  Phase  III  -  Integration  of  deficiency  liata  acroea  mission  areas. 

(see  Table  II) 

The  prioritiaed  lists  of  major  deficiencies  obtained  in  Phase  II  waa 
then  integrated  across  mission  areas.  TRADOC  conducted  four  separate  sessions 
with  general  officers  (CO)  from  proponent  schools  and  centers.  The 
composition  of  each  panel  reflected  a  broad  coverage  of  expertise.  Each  GO 
panel  then  integrated  the  12  mission  area  lists,  two  at  a  time,  using  the 
pairwise  comparisons  technique.  Each  GO  panel  member  was  asked  to  consider 
the  top  ranked  deficiency  in  each  mission  area  list  and  compare  them  two  at  a 

time  to  determine  their  relative  importance  as  was  done  in  Phase  II.  The 

lowest  ranked  deficiency  in  each  list  was  similarly  compared.  Based  on  a 
frequency  count  of  theee  deficienciea,  the  order  of  integration  for  the 
miasion  area  liets  was  established  (see  figure  5).  The  list  judged  to  have 
the  single  most  important  deficiency  became  the  base  list  and  the  list  with 
the  eecond  most  important  deficiency  was  integrated  into  the  base  list.  The 
top  ranked  deficiency  in  the  second  list  will  be  fixed  on  the  base  list  in 
comparison  with  its  importance  to  the  other  deficiencies  on  the  base  list. 

Its  position  on  the  base  list  was  determined  by  the  consensus  of  the  GO 
panel.  The  lowest  ranked  deficiency  in  the  second  list  was  fixed  on  the  base 

list  in  the  same  manner.  The  interval  on  the  base  list  between  the  two  fixed 


deficiencies  form*  the  region  of  integration.  The  renaining  deficiencies  on 
the  second  list  were  then  mathematically  transformed  into  the  base  list.  The 
CO  panel  screened  the  resultant  list  for  any  incongruencies  or  major 
discrepancies  and  made  appropriate  adjustments.  Consensus  among  the  panel 
members  was  required  for  adjustments  to  be  made.  This  list  became  the  new 
base  list.  Using  the  above  procedure  which  is  graphically  portrayed  in  figure 
6,  the  next  ordered  list  was  merged  into  the  new  base  list  until  all  12  lists 
were  integrated.  Pour  prioritised  lists  of  major  deficiencies  across  mission 
areas  (one  from  each  panel)  emerged  from  this  phase. 

TABLE  II 

PHASI  III  -  INTIMATION  OP 
MPICIINCIIS  ACROSS  ALL  MISSION 
ARIAS 


•  piousn  ptiMmzn  deficiencies  -  rub  anead  mm 

•  ESTABLISH  F0Vt  M  FANILS-  HORN  ttVEIMI  OF 
EKPCRTISC 
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•  INTIMATE  MR  DEFICIENCIES  Util  — TWO  LISTS  IT  R  TIM 
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PNASI  IN  (00  SISSION) 


■  I 

■• 

x 

fA 

'a 

m 

i — 

m 

■ 

• 

8 

«  in- 

> 

«  u> 

■  1+1  ■• 


FIGURE  6 

d.  Phaaa  IV  -  Aggragation  of  tha  four  GO  panal  raaulta, 

A  fifth  GO  panal  vaa  convanad  at  BQ  TRADOC  to  aggragata  tha  raaulta  of 
tha  othar  GO  aaaaioaa.  Thia  panal  eonaiatad  of  ganaral  offlcara  from  BQ 
TRADOC,  Combinad  Ana  Cantor  (CAC),  Ft  Laavanvorth,  K8,  Logiatica  Cantar 
(LOGC),  Ft  Laa,  VA,  Soldiar  Support  Cantar  (S8C),  Ft  Ban  Harriaon,  IN,  and  tha 
US  Any  Forcaa  Command  (FOR8COM),  Tha  four  intagratad  liata  of  major 
dafieianeiaa  war a  raviavad  and  mathamatically  margad  by  aggragatlng  tha 
pairviaa  compariaon  raaulta  froa  tha  four  GO  aaaalona.  Tha  aggragatad  top  and 
bottom  dafieianeiaa  aatabliahad  tha  ordar  of  marga  and  tha  ragion  of 
intagration  during  thia  procaaa.  Thia  liat  vaa  aeraanad  for  diaerapanciaa  and 
final  adjuatmanta  vara  mada  baaad  on  tha  conaanaua  of  tha  panal,  Fhaaa  IV  vaa 
tha  final  atap  in  tha  ovarall  prioritiaation  affnrt,  tha  raault  baing  a  aingla 
prioritiaad  liat  of  major  dafieianeiaa  acroaa  all  TRADOC  miaaion  araaa  (aaa 
figura  6). 


PHASE  IV 

AOORIfATID  IN  TRADOC  LIST 


'k'fc 


FIGURE  7 
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The  TRADOC  integration  and  prioritiaation  proceae  occura  annually.  The 
currant  methodology  la  new  and,  hence,  aaka  for  inprovenente  in  overcoming 
revere 1  ahortfalla.  The  technique  doea  not  allow  for  weighting  either 
individual  deficienciea  or  miaaion  area  liata  to  give  conaideration  to  the 
fact  that  not  all  mieaion  areaa  are  equally  deficient  nor  equally  aignif leant. 
Additionally,  an  artificial  ceiling  of  20  deficienciea  per  miaaion  area  had  to 
be  eatabliahed  to  limit  the  numbera  of  required  comparieona  to  an  acceptable 
level.  In  reality,  many  MAAa  produced  deficienciea  numbering  in  the  hundreda. 

Contributora  deairing  additional  information  or  wiahing  to  comment  on 
propoaed  improveoente  to  the  integration  and  prioritiaation  proceea  are 
encouraged  to  contact  the  author  by  phoning  (804)  727-3004  or  by  writing  HQ 
TRADOC,  ATTN:  ATCD-AM,  Ft  Monroe.  VA  23669. 
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ABSTRACT: 

— -;PWhen  a  definite  integral 


6Lvfhi  l 

jsannot  be  evaluated  exact  ly  ^"-w-e^tum  to 
computer-based  methoda  for  approximations.  There  are  many  different 
kinde  of  enoh  procedure*,  but/  they  divide  roughly  into  two  olaeaea: 
deterministic  and  random.  -14  explore  the  uae  of  methoda  that 
combine  these  two  approaohea,  preserving  the  unbiasedness  and  error 
estimation  advantages  of  random  method*,  bnt  at  the  same  time 
maintaining  the  closer  approximation  generally  found  in 
deterministic  methods. 


INTRODUCTION: 
Cons ider 


itimation  of  the  integral 


I 


f 


f (x )  dx . 


A  classical  deterministic  approaoh  might  be  to  try  Simpson's 
rule,  whioh  ueea  the  approximation 


j  [f (-1)  +  4f  (0)  +  f (1)1  . 


This  is  the  integral  of  the  interpolating  quadratic  approximation 
to  f  that  takes  on  the  same  values  at  -1.  0,  and  1.  The 
advantage  is  exactness  for  quadratics  (and  oubios  too.  by 
symmetry.)  The  disadvantage  of  this  approach  is  that  the  error 
assessment  involves  a  higher  order  derivative  of  the  funotion  f. 
which  may  be  difficult  to  find  (especially  for  a  funotion  whioh  was 
difficult  to  integrate  analytically  in  the  first  place.) 

A  second  approach  might  be  the  standard  random  method  known  as 
simple  Mont*  Carlo.  To  matoh  this  with  the  previous  approach,  let 


ua  also  do  thro*  funotion  evaluation*,  tad  estimate  the  integral  aa 


|  tf(X1)  +  f(I2)  +  f(i3>] 


where  the  faaotioa  ie  evaluated  at  three  valaee  that  are 
independently  and  unifornly  distributed  raadoa  variables 
la  (-1,  1).  The  advantage  of  this  approach  is  unbiasedness:  no 
aatter  what  f  ie,  eo  long  ae  it  ia  iategrable,  the  average  (in  the 
eenae  of  expectation)  of  this  approziaation  will  be  the  trae 
integral  of  f.  The  praotioal  advantage  of  this  is  that  the 
approziaation  oan  be  repeated  several  tlaea,  generating  new 
independent  randoa  variables  each  tine,  and  froa  these  resalta  the 
average  and  standard  error  oan  be  calculated.  This  givea  an  error 
aaaeaaent  without  need  for  further  aatheaatioal  analysis  (whioh 
aight  not  be  traotable.)  The  disadvantage  of  this  approaoh  la  that 
it  is  generally  not  as  ezaot  as  Siapson's  rule  and  the  higher  order 
deterainistio  aethods. 

Methods  that  ooabine  the  advantages  (but  not  the  disadvantages) 
of  each  aethod  have  been  available  sinoe  the  work  of  Sraakov  and 
Zolotukhin  (1960).  This  and  other  randoa  integration  procedures  are 
discussed  by  Haaaereley  and  landaooab  (1964).  tainted  work  has  been 
done  by  Hogues,  Corbett,  add  Patterson  (1981),  Cranley  and 
Patterson  (1970,  197 6),  Haber  (1969,  1970),  and  Quaokenbush  (1969). 
Ve  have  proposed  (Siegel  and  Zaabuto,  1982)  the  use  of  syaaetrio 
quadrature  deaigne  of  2k+l  points  whioh  aohieve  unbiasedness 
together  with  exactness  for  polynoaials  of  degree  2k+l  for  this 
problea. 


THB  THBEB- POINT  INTEGRATION  RULE: 

he  syaaetrio  randoa  unbiased  3-point  integration  rule  is 
unique,  and  ia  given  by 


I,({)  -  —^7  tf<-?>  +  2(3<2-l)f(0)  ♦  f($)J 

1  Jr 


where  {  has  the  distribution  of  the  cube  root  of  a  uniforaly 
distributed  randoa  variable  in  (0,  1). 
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This  may  be  thought  of  a*  a  systematically  randoa  (aa  oppoaad  to 
a  completely  randoa)  adaptation  of  Siapaon'a  rale.  The  function  la 
evaluated  at  three  points:  0,  and  ( .  in  a  aymnatric  bat 

partially  randoa  design.  The  weights  are  ohosen  properly  in  order 
to  assure  that  the  approxiaati on  is  equal  to  the  integral  of  the 
quadratic  function  that  agrees  with  f  at  these  three  points;  this 
assures  exactness  for  quadratics  (aa  does  Simpson's  rule.)  Finally, 
there  reaaics  a  degree  of  frsedoa:  £  aust  be  specified.  As  it 
turns  out,  there  is  only  one  distribution  for  (  that  will  preaerwe 
the  unbiasedness  property  of  staple  Monte  Carlo:  the  oube  root  of  a 
unifora.  It  is  like  applying  Siapson's  rule  with  a  speoial  randoa 
scaling  of  the  dosign  points  at  which  the  funotion  is  evaluated. 
Siapaon'a  rule  represents  only  one  extreme  ( <J  ■  1)  of  this 
oontinuua  of  possibilities  which  aust  be  aaapled  carefully  in  order 
to  obtain  an  unblasod  estiaate. 


BXAMPLB: 

Consider  the  integral 


If  -  j  oos  dx  -  £  -  1.27324  ... 


The  Integral  oan  be  evaluated  exaotly  in  this  case,  but  we  will  ube 
numerical  methods  in  order  to  gain  Insight  into  the  way  the 
procedures  work.  This  funotion  goes  from  aero  to  one  and  baok  to 
seroj  hence  the  siaple  Monte  Carlo  estimates  can  range  froa  0  to 
2.  The  syaaetric  randoa  deaiga  for  this  problea  yields  the 
following  integral  estiaate: 


I,<0  -  2  -  — ~  (1  -  oos  (4)1 

f  3{2  2 


This  la  a  much  flatter  function!  It  takes  on  values  froa 
1.1773...  to  1.3333...  When  these  values  are  sampled  randomly  it 

is  clear  that  they  will  fall  auoh  oloser  to  the  true  value 
(1.27324...)  than  siaple  Monte  Carlo  would.  Using  the  oube  root  of 
a  unifora  for  {  guarantees  that  the  results  will  be  oorreot  on  the 
average,  and  that  any  standard  error  or  confidence  interval  will  be 
aayaptotioally  oorreot.  Simpson's  rule,  representing  (  ■  1,  always 
yields  the  estiaate  1.333...,  one  of  the  extremes  that  the  randoa 
rule  oan  aohieve  (although  in  general,  Siapson's  estiaate  need  not 
be  an  extreae.) 
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TWO  DIMENSIONS: 

Thor*  1*  *  corresponding  ainpla  fornula  for  th*  two-d inan a i ona 1 
intagra 1 


f(*.ti)  dC  dt) 

-1  -1 


givan  by 


f(t*ii)*f<-ii.C>+f(w.-C>+f <-« .-n)  -  4f ( o»o) 

J#(C.H)  -  - * - r -  +  4 f  (  0 ,0 ) 

3(?a  +  0/2 


Thia  fornula  providaa  an  azaot  anavar  if  f  la  a  polynoaial  in  ( 
and  n  of  dagraa  at  aoat  3.  It  will  giv*  an  nnbiaaad  aatiaata  if 
U»H>  i*  aaaplad  froa  tha  density 

(C*  +  ?  and  n  in  (o.i). 
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ABSTRACT.  A  measure  of  force  effectiveness  that  Is  often  used  In  military 
analysis  of  combat  Is  the  Loss  Exchange  Ratio  (LER).  In  many  cases,  the  Input 
required  to  calculate  this  measaure  Is  obtained  by  replicating  a  stochastic 
wargame  model  by  using  a  computer  or  a  manual  exercise.  It  would  be  useful  to 
determine  a  confidence  Interval  about  this  measure  of  force  effectiveness.  This 
confidence  Interval  would  enable  the  analyst  to  examine  problems  concerning  the 
precision  of  the  measure,  and  compute  the  replication  requirement  for  a  stated 
degree  of  precision.  Hypothesis  testing  could  be  done  to  compare  the  LERs  of 
different  alternative  weapon  systems  Introduced  Into  the  force.  Two  methods  of 
solution  are  developed  and  proposed  and  an  example  Is  given,  f - - — —  _ 
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II .  Introduction.  A  measure  of  force  effectiveness  that  Is  often  used  In 
military  analysis  is  the  Loss  Exchange  Ratio.  This  measure  Is  defined  as  the 
ratio  of  red  casualties  (R)  to  blue  casualties  (B): 

LER  -  R/B  (1) 

The  LER  shows  an  operational  advantage  to  the  blue  force  If  R  >  B.  For  this 
discussion  the  values  of  R  and  B  are  obtained  by  replicating  a  stochastic 
wargame  model.  For  each  replication  of  the  wargame  the  parled  values  of  R  and  B 
are  recorded.  The  average  LER,  (LER),  Is  computed  as: 


LER  -  _R_  (2) 

B 

Because  the  generators  of  these  average  values  are  the  results  of  a  stochastic 
wargame,  It  would  be  useful  to  determine  a  confidence  Interval  about  the 
measure.  The  confidence  interval  could  be  used  to  answer  the  following 
questions: 

1.  Is  the  LER  >  1? 

2.  What  sample  size  Is  required  to  estimate  the  LER  with  some  stated  degree 
of  precision? 

3.  Are  various  measures  of  LER  statistically  different  from  each  other  at 
some  selected  level  of  significance? 

In  order  to  determine  the  confidence  Interval  for  the  LER  It  Is  necessary  to 
compute  the  variance  of  the  estimate. 

III.  Error  Propagation.  It  Is  well  known  (reference  Beers)  that  If  a 
function  of  the  form 

y  g  f(x^,  ^2 t  •  «•»  x^)  (3) 

exists,  then  the  variance  of  this  function  can  be  written  as: 


var(y)  -  )2varUi)  +  flj  <|~~) (— (var(Xl))1/2(var(Xj))l/2  (4) 

tjj  1  j 

where  ^ 

var(xj)  ■  variance  of  the  1w  variable 

a  a  w 

R^j  ■  correlation  between  the  1  and  J  variable. 

The  general  form  of  the  LER  Is  shown  as  equation  (2);  therefore,  the  variance 
of  this  form  c«.n  be  written  as: 

VAR  (LER)  •  VAR  (~)  (5) 

6 


Applying  equation  (4)  to  compute  the  variance  of  the  (LER)  we  obtain  the 
following: 


Var(LER)  -  U/B)2var<R)  +  <-R/§2)2var(B)  +  2(l/6)(-R/B2)Rj^4ar(R)/var(I)  (6) 
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The  following  Is  also  true: 

2 

VAR(R)  3  S  R/n;  where  n  3  the  number  of  replications 
VAR(B)  3  S2B/n 


>  (7) 


Rf^  B  3  RR  B’  as  Pointe<1  out  ’n  Beers  (Reference  1,  page  31)  J 

Substituting  the  set  of  relationships  (7)  into  equation  (6)  gives  an  expression 
for  YAR(LER'). 


VAR (LER)  -  ^(1/B)2S2  +  (-r/B2)2s|  +  2(1/B)(-R/B2)RSRSB] 


(8) 


This  expression  is  equivalent  to  the  one  given  by  Fishman  (Reference  3,  page  59) 
for  "large"  value  of  n.  The  appropriate  100(1  -  ot  )  confidence  Interval  (C.I.) 
for  the  LER  could  be  computed  as: 


100(1  -  a  )  C.I.  (LER) 


LER  +  t 


/ 


VAR (LER) 


(9) 


Where  t  is  Student's  t  with  (n  -  1)  degrees  of  freedom.  Equation  (9)  Is 
supported,  in  part,  by  Rao  (Reference  5,  pages  319  through  321)  who  points  out 
that  In  practical  applications,  distributions  of  the  form  we  are  studying  are 
asymptotically  normal  if  the  co-variance  and  partial  derivatives  are  continuous. 

IV.  Fieller's  Theorem.  Goldstein's  text  Biostatistics  (reference  4,  page  184) 
gives  a  sTtuaOon'  where  It  is  necessary  to  compute  a  confidence  interval  for  a 
ratio.  The  problem  from  the  Goldstein  text  Is  given  below.  Note  the 
correspondence  between  the  biological  experiment  and  the  force-on-force 
slmulatl on. 

"Quite  often  in  biological  experimentation  one  wishes  to  estimate  a  ratio 
from  a  set  of  observations  on  the  numerator  (y)  and  another  set  of  observations 
on  the  denominator  (x).  Now  these  may  be  paired  observations,  each  Item  in  a 
sample  supplying  a  value  of  y  and  a  value  of  x,  so  that  there  may  be  some  degree 
of  correlation  between  the  two. . .Suppose  the  protein  content  of  cells  per  unit 
of  DNA  is  to  be  determined.  If  the  cells  In  question  are  growing  In  replicate 
bottles,  we  may  determine  both  DNA  and  protein  on  the  contents  of  each 
bottle.. .In  (this)  case,  replicate  estimates  of  the  desired  ratio  will  be 
available.. . " 

Using  this  example,  the  replicate  bottles  become  the  replicated  force-on-force 
simulations  and  the  correlated  values  of  both  ONA  and  protein  within  each 
replicate  bottle  become  the  number  of  red  and  blue  casualties  respectively. 
Goldstein  then  points  out,  "that  the  appropriate  limits  of  the  true  ratio  R 
whose  estimate  is  y/x,  are  given  by  Fieller's  Theorem  as  roots  of  a  quadratic 
equation: 


359 


no) 


-  [x2  -  t2(S2/n)jR2  +  2[xy  -  t2rSxSy/n]R  -  [y2  -  t2(S2/n)]  ■  0 


where  r  ■  sample  correlation  coefficient 

n  ■  number  of  paired  observations  In  the  sample 

2  2 

S  x,  S  y  •  sample  variances 

t  »  two-tailed  value  of  Student's  t  w1th(n  -  l)degrees 
of  freedom" 

Finney  (reference  2,  page  27  through  29)  gives  a  discussion  of  Fleller's  Theorem 
and  Its  application  to  finding  fiducial  limits  to  a  ratio  of  two  means.  Solving 
equation  (10)  for  R  and  using  the  notation  consistent  with  the  LER  and 
force-on- force  simulation  (l.e.,  x  =  B  and  y  s  !)  we  obtain: 


tirSBSR 


'(BR  - 


tIrSBSR.2 


tn4-> 


s2  s2 

-  [B2  -  t2(-jj)][R2  -  t2<-*)) 


Fishman's  (reference  3,  pages  5S  through  61)  section  on  confidence  Intervals 
uses  a  quadlstlc  form  similar  to  equations  (10)  and  (11)  to  derive  a  confidence 
Interval  for  R  which  Is  cited  as  work  done  by  Fleller  and  Is  suggested  for  use 
In  simulation  by  Crane  and  lglehart. 

V.  Example.  The  following  numerical  example  Is  based  on  the  data  used  by 
Goldstein  for  his  explanation  of  replicate  bottles  of  DMA  and  protoin  cl  tod  In 
section  IV  of  this  paper.  Tho  column  headings  have  been  changed  to  red  and  blue 
casualties,  respectively. 


TABLE  1. 


Red  and  Blue  Casualties  for  each  Battle  Replication 
(adopted  from  Goldstein) 

Replication 

Number  Red  Casualties 

Number  Blue  Casualties 

1 

12 

5 

2 

14 

7 

3 

12 

3 

4 

12 

3 

5 

13 

8 

6 

13 

6 

7 

13 

4 

The  suninary  statistics  are: 


N  -  7 

R  ■  12.71  casualties 
Sg  *  .756  casualties 
B  -  5.14  casualties 
Sg  c  1.952  casualties 
R  ■  .710 

and  the  eslmate  of  LER  (LER)  -  2.47. 

a.  Error  Propagation.  Applying  these  data  to  equation  (8)  gives  a  value 
of  the  VAR  (LER)  equal  to  .101.  The  appropriate  value  of  t(for  a  95%  confidence 
Interval  with  6  degrees  of  freedom)  Is  2.447.  The  95%  confidence  limits  for  the 
LER  (using  equation  (9))  are  3.3  and  1.7.  Note  that  the  lower  confidence  limit 
(1.7)  Is  greater  than  0. 

b.  Fleller's  Method.  Applying  these  same  summary  data  to  equation  (11) 
and  solving  for  both  roots  of  the  quadlstlc  gives  values  of  3.7  and  1.9, 
respectively.  Note  that  the  lower  confidence  limit  (1.9)  Is  also  greater  than 
0. 


VI.  Conclusions.  Although  this  effort  represents  a  limited  study,  the 
following  conclusions  are  emerging. 

a.  Error  Progatlon  and  Fleller's  method  appear  to  give  "reasonably" 
consistent  results. 

b.  Fleller's  method  Is  the  preferred  way  to  compute  a  confidence  Interval 
about  a  ratio.  This  conclusion  Is  based  on  some  of  the  existing  literature 
(Finney,  Fishman,  and  Goldstein). 

c.  The  error  propagation  method  should  Increase  In  accuracy  as "n "becomes 
large  (Fishman  and  Rao);  however,  Fleller's  Theorem  should  be  more  appropriate 
for  the  smaller  numbers  of  replications  that  are  used  In  force-on-force 
simulation. 
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AN  EXAMPLE  OF  SOFTWARE  VALIDATION  USING  A  FACTORIAL  DESIGN 
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ABSTRACT 


This  paper  reports  on  the  effects  of  the  Department  of  Energy  to  test 


and  evaluate,  validate  ,  a  large  computer  model,  which  represents  the  world 


petroleum  distribution  system.  The  evaluation  technique  employed  Is  a 
complete^' factorial  design.  The  main  effects,  as  well  as  all  2nd  order 
effects  are  estimated.  The  technique  provided  crlterlal  Insights  Into  the 
nature  of  the  software  Identifying  errors  and  assisting  In  the  development,  of 
a  methodology,  for  reexamining  candidate  crude  mixes,  of  oil  stored  by  the 
government,  for  use  during  possible  petroleum  Interruptions. 


\ 


DISCLAIMER 

The  assumptions,  procedures,  analysis,  conclusions,  and  recommendations 
contained  In  this  paper  are  solely  those  of  the  author  and  do  not  represent 
any  official  policy  of  the  Department  of  Energy  or  U.S.  Government. 
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A1  Factorial  Design  Methodology 


An  experiment  was  performed  to  aeasure  the  affect  of  four  sate  of 
Input  factors  on  the  oil  market,  as  represented  by  a  large  linear 
prograalng  model  supplied  by  a  private  contractor.  Two  levels  for  each 
set  of  input  factors  wars  chosen  and  all  16  possible  combinations  of 
thaae  Input  factors  wars  used  as  aodel  input  to  the  modal.  This 
procedure,  a  2*  factorial  design  was  chosen  since  it  is  economical, 
easy  to  use  and  provides  a  great  deal  of  valuable  information. 
Specifically  a  two  (2)  level  factorial  design  has  the  following 
advantages: 

1.  If  sets  of  input  factors  are  varied  one  set  at  a  time  with  the 
remaining  factors  held  constant,  it  is  necessary  to  assume  that  the 
effect  would  be  the  same  at  other  settings  of  the  other  sets  of  input 
factors.  Factorial  designs  avoid  the  assumption. 

2.  If  the  effects  of  input  factors  act  addltivaly,  a  factorial 
design  estimates  those  effects  with  more  precision.  If  the  effects  of 
the  input  factors  do  not  act  addltivaly,  factorial  designs  can  detect 
and  estimate  the  interactions  which  measures  the  non-*additlvlty. 

3.  Factorial  designs  require  relatively  few  runs  per  set  of 
Input  factors  studied  and  can  Indicate  major  trends  and  determine 
promising  direction  for  further  investigation.  To  obtain  the  same 
precision  of  the  estimate  of  the  effects  measured,  in  this  effort  forty 
runs  would  have  had  to  be  run  using  the  traditional,  one  factor  at  a 
tlma  approach  rather  than  the  sixteen  used  in  the  experiment. 

4.  If  a  more  thorough  local  exploration  is  needed,  it  cen  be 
suitably  augmented  to  form  composite  designs. 

5.  These  designs  and  their  corresponding  fractional  designs  may 
be  used  as  building  blocks  so  that  the  degree  of  complexity  of  the 
finally  constructed  design  can  match  the  sophistication  of  the 
problem. 

To  perform  a  2^  factorial  design  the  two  levels  (or  versions)  for  four 
(4)  sets  of  input  factors  were  selected  and  all  sixteen  (16)  possible 
combinations  were  executed.  The  four  sets  of  Input  factors  and  their 
levels  (or  versions)  are  listed  on  the  following  page. 


Input  Factor 

1,  Composition  of  tha  8FR 


Laval a 

la,  Tha  SPR  la  f Iliad  with 
1001  light  and  swaat  crude 
represented  by  Ekoflsk 
(Sweet). 

lb,  Tha  SPR  la  f Iliad  with 
1001  heavy  and  aour  crude 
represented  by  Arab  Heavy 
(Sour). 


2,  Crude  pattern  or  availability  2a,  Historical  1976  BAU  case 

(SAU) 

2b,  As  2a  above  with  a  501 

closure  of  the  Persian  Gulf 
with  a  uniform  SPR  drawdown 
of  3MMBD  (501  P.G.)* 


3,  Refinery  configuration 


4,  Product  price  elasticities 


3a,  Worldwide  1978  refinery 
capacity  (1978) 

3b,  Worldwide  estimated 

1985  refinery  capacity 
(1983) 

4a,  The  proposed  set  of 

elasticities  compiled  for 
this  project  by  the 
contractor  (CON) 

4b,  As  4a  except  a  quite 
different  set  of 
elasticities  developed  by 
an  alternate  contractor 
for  major  products  in  the  US 
(ALT) 


*  The  selection  of  this  crude  oil  disruption  does  not  represent  the 

policy  of  the  Department  of  Bnergy  and  was  used  solely  to  evaluate  1 

the  reaction  of  the  model  to  changes  in  the  world  crude  pattern.  I 
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These  input  factors  combine  to  produce  the  following  design  or 
validation  matrix: 


Design  Matrix 


RUN 

SPR 

CRUDE 

REF 

NUMBER 

COMP 

PATTERN 

CONFIG 

ELAS 

1 

la 

2a 

3a 

4a 

2 

la 

2s 

3a 

4b 

3 

la 

2a 

3b 

4a 

4 

la 

2a 

3b 

4b 

5 

la 

2b 

3a 

4a 

6 

la 

2b 

3a 

4b 

7 

la 

2b 

3b 

4a 

8 

la 

2b 

3b 

4b 

9 

lb 

2a 

3a 

4a 

10 

lb 

2a 

3a 

4b 

11 

lb 

2a 

3b 

4a 

12 

lb 

2a 

3b 

4b 

13 

lb 

2b 

3a 

4e 

14 

lb 

2b 

3a 

4b 

15 

lb 

2b 

3b 

4a 

16 

lb 

2b 

3b 

4b 

Table  A-l 

The  lnterpertation  of  the  runs  in  Table  A-.1  Is  easily  Illustrated 
by  run  number  6  which  assumes  that  the  SPR  is  filled  with  100%  light 
and  sweet  crude  cil.  There  is  a  major  oil  interruption  consisting  of 
a  501  closure  of  the  Persian  Gulf  and  the  SPR  in  being  withdrawn  at 
the  rate  of  3MMBD  to  reduce  the  effects  of  the  crude  oil  shortfall. 

The  refinery  configuration  during  this  period  represents  the  1978  time 
frame.  Finally  the  set  of  elasticities  for  the  demand  of  products, 
within  the  US  developed  by  the  alternate  contractor  (ALT)  are  assumed.  The 
other  fifteen  runs  are  lntarperted  simllarlly. 


The  sixteen  runs  of  the  design  matrix,  may  be  visualised  geometrically  as  two 
cubes.  One  possible  visualisation  appears  in  figure  A-l  on  the  following  page. 
The  run  number  la  at  each  vertex. 
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A2  Calculation  of  Main  Effects 

Tha  "main  affact"  of  a  aat  of  input  factora  la  the  change  in  the 
response  as  we  nova  from  the  "a”  case  to  the  "b"  case  version  of  that 
eat  of  input  factora.  To  examine  the  effect  of  the  composition  of  the 
8 PR  a  table  of  eight  pair  of  colum  vectors  was  constructed  (aee  table 
A-2)  •  Aside  from  experimental  error,  the  difference  between  the  first 
column  vector  of  the  pair  and  the  second  column  vector  in  the  pair  is 
due  to  a  change,  in  the  composition  of  tha  JPR.  The  average  of  these 
eight  differences  (one  difference  for  each  pair  of  column  vectors)  Is 
tha  main  affact  due  to  the  composition  of  the  8PR.  Table  A-2  contrasts 
tha  composition  of  tha  reservs.  If  the  columns  are  rearanged  so  that 
the  run  msabsrs  are  in  a  as sending  order,  one  obtains  tha  tablo 
contrasting  the  product  price  elasticities.  Similar  rearangements 
yield  tables  contrasting  the  other  two  seta  of  input  factor. 

Geometrically  speaking,  using  Figure  A-l  the  main  effects  are 
calculated  from  corresponding  vertices  from  tha  two  cubes  as  described 
below. 


Input  factor 

Composition  of  8PR 


Crude  pattarn/avallablllty 


Refinery  configuration 


Left  side  of  both  cubes  vs 
the  right  side  of  both 
cubes 

The  front  of  both  cubes 
vs  tha  backs  of  both 
cubes 

The  bottom  of  both  cubes 
vs  the  tops  of  both  cubes. 


Product  price  elasticities 


The  left  cube  vs  the  right 


5. A3  2nd-0rder  Interaction  effects 


Suppose  that  one  le  Intareeted  In  examining  the  effects  of  two  eete  of 
Input  factor*;  for  exaapla,  refinery  configuration  and  crude  pattern. 
Then  the  sixteen  runs  of  the  factorial  design  can  be  grouped  into  four 
sets  of  four  runs  each.  Each  run  in  the  group  would  have  the  sane 
value  for  the  input  factors  studied,  although  other  input  factore  would 
vary  within  aach  group.  Assume  that  for  the  business  as  usual  crude 
pattern  with  the  1978  refinery  configuration,  the  average  value  for  the 
output  variable  being  studied  is  100.  This  will  be  the  baa*  point. 

Also  assume  that  the  main  effects  for  the  crude  pattern  and  the 
refinery  configuration  are  25  and  10  respectively.  This  means  that, 
on  the  average,  changing  from  a  BAU  crude  pattern  to  a  502  closure  of 
the  Persian  Gulf  will  Increase  the  output  variable  under  study  by  25. 
Likewise  a  change  from  the  1978  refinery  configuration  to  the  1985 
refinery  configuration,  will  on  the  average,  Increase  this  asms  output 
variable  by  10.  If  the  input  factors  act  addltivaly,  then  the  average 
value  of  the  output  variable  with  both  the  1985  refinery  configuration 
and  a  50  percent  closure  of  the  Persian  Gulf  would  be 
100  +  25  +  10  -  135. 

This  artificial  case  is  represented  by  the  upper  diagram  in  figure  A-2. 
Note  that  the  quantity 

(b  +  c  -a  -d)/2  -  (110  +  125  -100  -135)/2  -  0 
l.e.  there  is  no  Interaction. 

Suppose  that  the  input  factors  do  not  act  addltivaly,  and  the  base 
point  of  100  and  main  effects  ars  the  same.  Then  the  resulting 
measurements  could  be  described  by  the  lower  diagram  in  figure  A-2. 

The  input  factors  are  now  said  to  Interact.  By  convention  a  measure  of 
this  interaction  la 

(b  +  c  -a  -d)/2-(145  +  160  -100  -135)/2  -  35 

This  is  the  second  order  Interaction  and  is  called  the  refinery 
configuration  X  crude  pattern  Interaction. 

Like  the  main  affect,  the  2nd  order  interaction  is  the  difference 
between  two  averages,  eight  of  the  sixteen  results  being  Included  in 
one  average  and  eight  in  the  other.  Analogous  explanations  are  easily 
constructed  for  all  other  2nd  order  interaction  effects. 

5.A4  Higher-Order  interaction  effects  and  the  Standard  Error. 

Similar  procedures  to  thoaa  above  can  be  given  for  deriving  the  third 
and  fourth-order  interactions.  Due  to  the  similarity  of  response 
functione  it  is  reasonable  to  assume  that  higher-ordered  interactions 
are  negligible  and  measure  differences  erlslng  principally  from 
experimental  error.  Thus  the  mean,  of  the  sum  of  squares,  of  these 
Interactions  give  an  estimated  value  for  the  variance  of  an  effect, 
having  five  degrees  of  freedom.  The  square  root  of  this  value  le  an 
estimate  of  the  standard  error. 

The  level  of  statistical  significance  chosen  for  this  study  was  p-.Ol. 
In  order  to  select  the  statistically  significant  main  effects  and 
second  order  interactions  multiply  the  standard  error  by  tl-p/2*4.032 
any  main  effect  or  interaction  greater  than  this  product  is  considered 
statistically  significant. 
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INTERPRETATION  OT  2-MAY  DI 


PATTERN 


3.B1  Summary  of  significant  effects 


T«bl«  B-0  on  cho  following  page  is  a  summary  of  all  daCactad  aaln  affacts  and 
2nd  order  lntaractiona  which  warn  significant  at  tha  p»0.01  level.  In  addition 
■actions  B2|  B3  and  14  provida  datailad  anal vs Is  of  tha  inport  bill  and  total 
product  consumption.  Tha  analysis  of  tha  raiaainlng  output  variables  la 
itrsl|ht  forward  and  available  froa  tha  author  upon  request.  Each  coltnn  and 
■ubaactlon  of  tha  analysis  reports  on  tha  univariate  analysis  of  variance  of 
tha  aalactad  output  variable.  No  attempt  wae  made  to  perform  a  multivariate 
analysis  of  variance  since  model  Is  completely  deterministic  and  not 
stochastic. 


3.11  Sunary  of  significant  tfftcta 

i 

Tabla  B-0  oo  the  following  page  Ir  a  iiaaary  of  all  detected  main  affacta  and 
2nd  ordar  lntaraetlona  which  ward  significant  at  tha  p-0.01  level.  In  addition 
sactlona  12.  B3  and  B4  provide  detailed  analyala  of  tha  laport  bill  and  total 
product  consusptlon.  The  analysis  of  tha  renaming  output  varlablee  la 
straight  forward  and  available  froa  the  author  upon  request.  Each  colunn  and 
subsection  of  tha  analysis  reports  on  tha  univariate  analysis  of  variance  of 
tha  selected  output  variable.  No  attanpt  waa  aade  to  perforo  a  aultlvarlate 
analysis  of  variance  since  nodal  la  coapletaly  deterministic  and  not 
stochastic. 


? 


B2  Analysis  of  tha  U.S.  la  port  'oil! 


This  aaction  hat  baan  included  to  demonstrate  tha  usa  of  factorial  designs 
to  find  arrora  within  coaplax  computer  models.  A  changa  of  $21.0  billion 
in  tha  U.S.  Import  bill  due  to  a  change  in  refinery  configuration  la 
unraaaonabla.  Thla  result  prompted  an  lnveatlgatlon  Into  tha  method  of 
calculating  the  U.S.  Import  bill  and  tha  correction  of  the  appropriate 
coda.  Tha  raault  of  the  revlaed  U.S.  Import  bill  Immediately  follow  this 
aaction. 


Thera  teemed  to  be  appreciable  2nd  order  intaractlona  between  product 
price  elasticities,  refinery  configurations,  and  tha  crude  pattern  when 
tha  model  eetlmatee  tha  U.S.  Import  bill.  Therefore,  tha  flrat  two  aata 
of  Input  factora  had  to  be  evaluated  jointly  with  the  crude  pattarna. 

Tha  two-way  diagrams  of  figure  ^1  indicate  tha  nature  of  thaae 
Intaractlona. 


During  a  50  percent  cloaura  of  tha  Persian  Gulf  with  a  uniform  drawdown 
of  3  MMBD  from  the  SPR,  tha  modal  estimated  that  If  tha  ALT  sat  of 
elasticities  are  correct,  rather  than  those  compiled  by  the  contractor 
(CON),  the  Import  bill  will  be  $11,297  million  less.  The  change  in 
product  price  elasticities  had  no  affect  on  the  original  estimate  of  the 
U.S.  Import  bill  In  the  BAU  case. 


In  a  business  as  usual  environment,  the  modal  estimates  that  a  change  In 
the  refinery  configurations  from  the  1978  configuration  to  the  estimated 
configuration  In  1985  Increases  the  Import  bill  by  $10.9  billion.  With  a 
50  percent  closure  of  the  Persian  Gulf  and  a  uniform  drawdown  of  the  SPR 
of  3  MMBD,  the  affect  of  the  change  in  refinery  configuration  widens  to 
$21.0  billion  and  all  levels  ere  lower. 


The  SPR  composition  did  not  have  a  statistically  significant  effect  when 
the  model  eslmated  the  U.S.  Import  bill  over  the  levels  of  Input  factors 
tested. 


U.S.  import  bill  in  millions  of  dollars 


Main  Effects  astimata 

Product  price  alasticitias  -5648  * 

Rail nary  configuration  15971  * 

Crude  pattsrn/avallability  -7340  * 

Composition  of  the  SPR  556 

2nd  Order  Interactions  astimata 

Product  prlca  elasticities  X 

Refinery  configuration  -902 

Product  prlca  alasticitias  X 

Crude  pattern/availablllty  -5648  * 

Product  prlca  alasticitias  X 

Composition  of  the  SPR,  1258 

Refinery  configuration  X 

Crude  pattern/availablllty  5041  * 

Refinery  configuration  X 

Composition  of  the  8PR  839 

Crude  pattern/availablllty  X 

Compoaitlon  of  the  SPR.  563 

estimated  standard  error  787 

level  of  etatistlcal  significance  3146 

at  p  less  than  ,01 


*  significant  affects  at  p  lass  than  .01 

Table  1-3 
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B3  Reviled  Analysis  of  the  U.S.  import  bill 


The  prevloue  conclusions  of  the  import  bill  relied  some  eerloue  questions  on 
the  technlquei  used  within  the  model  to  eielmete  the  bill.  Further 
Investigations  lead  to  the  dlacovery  of  an  error  within  the  aoftware  which 
wee  reaponalble  for  $21.0  billion  increaae.  The  error  hae  subsequently  been 
corrected  and  the  analyaie  of  the  corrected  veralon  of  the  Import  bill 
appeara  below. 

There  are  perceptible  2nd  order  Interactions  between  product  price 
elasticities,  the  composition  of  the  SPR,  the  refinery  configuration,  and 
the  crude  pet tern  when  the  revised  model  eatimatea  the  U.S.  Import  bill. 
Therefore,  the  first  three  sets  of  Input  factors  must  be  evaluated  Jointly 
with  the  crude  pattern.  Tha  two  way  diagrams  of  figure  B-2  Indicates  the 
nature  of  these  Interactions. 


During  a  30Z  closure  of  the  Persian  Gulf  with  a  uniform  drawdown  of  the  SPR, 
the  model  estlautes  that  If  the  ALT  elasticities  are  correct,  rather  than 
elasticities  sat  CON,  the  U.S.  Import  bill  will  be  $10.6  billion  less. 

Under  the  same  interruption,  the  effect  of  a  totally  aour  SPR  rather  than  a 
totally  sweet  SPR  will  Increase  the  Import  bill  by  $2  billion.  With  the 
business  as  usual  crude  patterns,  neither  the  change  In  product  price 
elasticities  nor  a  change  In  the  composition  of  the  SPR  has  an  affect  on  the 
estimated  U.S.  Import  bill.  The  model  estimates  that  upgrading  the  refinery 
configuration  decreases  the  Import  bill  by  $0*2  billion  with  a  business  as 
usual  crude  pettern.  During  an  Interruption,  this  same  upgrading  may  reduce 
the  Import  bill  by  $1.1  billion. 

In  comparing  the  revised  estimate  of  the  Import  bill  with  the  initial 
estimate  of  the  Import  bill,  two  changes  are  most  apparent.  First,  the 
Initial  estlmatas  of  tha  Import  bill  estimates  that  the  average  change  due 
to  upgrading  the  refinery  configuration  increased  the  Import  bill  by  $16.0 
billion.  An  Increase  beyond  rsaaon,  especially  when  one  expects  a 
reduction.  The  more  reasonable  result  of  an  average  decrease  of  0.6  billion 
was  estimated  In  tha  revised  run.  Secondly,  the  composition  o<  tha  SPR  does 
not  have  a  statistically  significant  effect  over  the  range  of  nput  factors 
tested  In  the  original  estimate  of  Import  bill,  but  it  does  In  the  revised 
version. 
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Revised  U.S.  leper t  bill  in  billions  of  dollers 


Mein  Effects 

Product  price  elasticities 
Refinery  configuration 
Crude  pattern/avallabllity 
Coepositlon  of  the  SPR 


2nd  Order  Interactions 

Product  price  elasticities  X 
Refinery  configuration 

Product  price  elasticities  X 
Crude  pattern/availability 

Product  price  elasticities  X 
Coepositlon  of  the  SPR. 

Refinery  configuration  X 
Crude  pattern/avallabllity 

Refinery  configuration  X 
Composition  of  the  SPR 

Crude  pattern/avallabllity  X 
Coepositlon  of  the  SPR. 

estimated  standard  error 

level  of  statistical  significance 

at  p  leas  than  .01 

*  significant  ef facte  at  p  leas  than  .01 

Table  B-4 


estimate 

-3.26  * 
-0.64  • 
18.01  * 
0.96  * 

estimate 

-0.19 

-5.26  * 

-0.11 

-0.44  * 

-0.01 

0.96  * 

.11 

.42 
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14  Analysis  of  ths  Total  U.8.  eoasuapeion  of  products 

Thors  ars  important  2nd  ordsr  lntsractions  between  elasticities,  tha 
composition  of  tha  SPR,  raflnsry  configurations,  and  tha  cruda  pattern 
which  affect  tha  estimates  of  total  consissptlon  of  products. 

Tbsrsfors,  each  of  tha  first  thrsa  sets  of  input  factors  aust  be 
evaluated  jointly  with  tha  input  factors  representing  tha  cruda 
pattern*  Tha  three  two-way  dlagraas  depicting  tha  nature  of  the 
interactions  appear  in  figure  B-3. 

During  a  50  percent  closure  of  ths  Persian  Gulf,  the  use  of  the  ALT 
elasticities  rather  than  the  CON  elasticities  decreases  the  estimated 
total  U.8.  consumption  of  products  by  933  MBD.  Under  the  ease 
interruption,  a  sour  8FB  produces  an  estimated  increase  In  the  total 
consumption  of  products  by  237  MBD  over  usa  of  a  sweet  8PR. 


When  using  the  8AU  crude  pattern,  the  model  estimates  that  neither 
variable  has  a  statistically  significant  affect  on  the  total  U.S. 
con e iap t ion  of  products. 

With  a  cruda  pattern  representing  the  BAU  case,  upgrading  the  refinery 
configuration  from  1978  to  1985  Increases  the  total  U.S.  consumption 
of  products  by  about  22  MBD,  whersas  the  same  upgrading  during  a  30 
percent  closure  of  the  Persian  Gulf  with  a  uniform  drawdown  of  ths  SPR 
of  3  MMBD  decreases  the  total  U.8.  consumption  of  products  by  about 
155  MBD.  This  is  the  only  example  in  the  study  of  a  crossed  pattern 
and  is  worthy  of  further  investigation.  This  crossed  pattern  is 
contrary  to  anticipated  results. 
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CRUOE  PATTERN 


The  U.S.  total  product  consumption  in  M4BD 


Main  Effacta  aatlmate 

Product  prlca  alaatlcltlaa  **466  * 

Raflnary  configuration  -67  * 

Cruda  pattern/ availability  -2817  * 

Composition  of  tha  SPR  118  * 

2nd  Ordar  Intaractlons  astlmata 

Product  prlca  alaatlcltlaa  X 

Raflnary  configuration  -20 

Product  prlca  alaatlcltlaa  X 

Cruda  pattarn/avallabllity  -466  * 

Product  prlca  alaatlcltlaa  X 

Composition  of  tha  SPR.  24 

Raflnary  configuration  X 

Cruda  pattarn/avallabllity  89  * 

Raflnary  configuration  X 

Composition  of  tha  SPR  -8 

Cruda  pattarn/avallabllity  X 

Composition  of  tha  8PR.  118  * 

aatlmatad  standard  arror  16 

laval  of  statlatlcal  significance  66 

at  p  la  as  than  .01 


*  significant  affects  at  p  less  than  .01 

Table  B-5 


Cl  Results 


Tha  corresponding,  results  of  the  experiment  were: 

a.  Slgnlf leant  differences  were  detected  in  the  United  8tetee  crude  oil  end 
product  import  patterns  associated  with  drawdown  of  the  two  SPR  types,  but 
relatively  little  differences  In  Che  level  and  pattern  of  product  demands 
supplied.  Of  the  products  examined,  only  the  consuaptlon  of  residual  fuel 
oil  was  significantly  effected  by  a  change  In  the  composition  of  the  8PR. 

b.  Significant  differences  In  all  exaalned  variables  were  detected  as  a  result 
of  changing  the  availability  of  crude  oil. 

c.  Chsnges  In  the  refinery  configuration  create  suall,  but  significant  effects 
la  several  aeasured  variables.  However, . changes  In  the  refinery 
configuration  should  not  effect  the  recommended  mix  of  crude  oil  to  be 
stored  In  the  reserve  since  the  2nd  order  refinery  configuration  X 
composition  of  the  SPR  Interaction  la  not  statistically  significant  for  the 
▼arlables  tested. 

d.  Significant  differences  In  the  pattern  of  US  demands,  for  several  products, 
resulted  from  the  application  of  tha  alternate  sets  of  product  demand  price 
elasticities.  However,  changes  of  the  elasticities  should  not  affect  the 
recommended  mix  of  crude  olla  to  be  stored  In  the  reserve  since  the  2nd 
order  elasticity  X  composition  of  the  8 PR  Interaction  is  not 
statistically  significant  for  the  variables  tested. 

Recommendation 


The  factorial  dealgn  detected  errors  within  the  US  import  bill  and  several 
othar  output  variables.  It  was  recommandad  that  this  modal  be  corrected  before 
It  was  put  Into  production  use. 

Post  Script 

In  addition  to  the  procedure  described  in  this  paper,  other  checks  of  the 
quality  of  the  software  were  made  Including  an  analysis  of  the  estimated 
results  for  the  year  197S  against  historical  data.  The  contractor  Incorporated 
the  recommended  corrections  and  In  the  fall  of  1982  the  corrected  modal 
produced  results  which  were  diametrically  oppocita  of  earlier  runs. 
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Nonparametric  Probability  Density  Estimation 
for  Data  Analysis  in  Several  Dimensions* 


David  W.  Scott 

Rice  University 
Houston.  TX  77251 


1.  Introduction 

- -  771  <_  v  f 

Our  purpose  Air  thia  paper  ia  to  illustrate  hov  nonparametric  proba¬ 
bility  density  estimates,  in  particular  the  corresponding  contour 
curves,  are  a  useful  adjunct  to  scatter  diagrams  when  performing  a  prel¬ 
iminary  examination  of  a  set  of  random  data  in  several  dimensions.  For 
a  preliminary  approach  we  generally  want  to  perform  fairly  simple  tasks 
with  free -form  techniques  to  uncover  structures  and  features  of  interest 
in  the  data.  Such  procedures  are  often  graphical  and  unlike  summary 
statistics  seldom  lead  to  smch  compression  of  the  data.  Tukey 
presents  a  wealth  of  such  procedures.  One  which  well  illustrates  the 
power  and  flexibility  of  these  preliminary  procedures  is  the  running 
median  smoothing  algorithm  for  time  series  data  (with  resmoothing  of  the 
rough  and  the  like).  Other  graphical  techniques  for  multivariate  data 
are  presented  in  Tukey  and  Tukey  (1981).  _ 

For  preliminary  viewing  of  one -dimensional  data,  both  scatter 
diagrams  and  frequency  curves  such  as  histograms  are  widely  and  success¬ 
fully  employed  to  examine  clustering,  tail  behavior,  and  skewness  of 

*This  research  was  supported  in  part  by  the  Army  Research  Office 
under  DAAG-29-82-K-0014  and  by  NASA/Lockheed  under  PO-0200100079. 
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data.  For  bivariate  data*  scatter  diagrams  are  in  practice  widely  pre- 
Terred  to  bivariate  frequency  curves.  Scatter  diagrams  of  three  dimen¬ 
sional  data  may  be  realized  by  viewing  a  projection  of  the  data  on  a 
rotating  plane  represented  by  the  screen  on  a  computer  graphics  termi¬ 
nal.  For  higher  dimensions  carefully  selected  projections  may  also  be 
viewed*  and  sophisticated  techniques  have  been  developed*  and  are  evolv¬ 
ing*  for  choosing  good  projections  (Friedman  and  Tukey*  1974). 
Apparently  the  success  of  frequency  curves  in  one  dimension  has  not 
readily  extended  to  higher  dimensions.  It  is  an  open  question  as  to  the 
number  of  dimensions  that  may  be  successfully  visualized  with  a  non- 
parametric  density  estimator  under  various  conditions  (sample  size*  for 
example).  It  is  our  purpose  to  illustrate  the  power  of  preliminary  fre¬ 
quency  curves  as  an  adjunct  to  viewing  scatter  diagrams. 

2.  Blvillata  Data 

He  shall  examine  a  data  set  which  contains  information  on  the 
status  of  the  coronary  arteries  of  371  men  suspected  of  having  heart 
disease*  having  experienced  episodes  of  severe  chest  pain.  These  data 
have  been  more  fully  described  and  analysed)  see  Gotto*  aL  a1*  (1977) 
and  8cott*  a!  Al»  (1978).  After  visual  examination  of  the  coronary 
arteries  by  angiography*  51  men  vere  determined  to  be  free  of  signifi¬ 
cant  coronary  artery  disease.  It  was  of  interest  to  compare  the  levels 
of  blood  fats*  plasma  cholesterol  and  plasma  triglyceride  concentra¬ 
tions*  between  the  group  of  51  disease-free  males  and  the  group  of  32b 
diseased  males.  The  scatter  diagrams  of  these  two  data  sets  are 
displayed  in  Figure  1.  Patients  with  elevated  levels  of  cholesterol  and 
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Figure  1.  Scatter  Diagrams 
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triglyceride  are  evident  among  the  diseased  males*  This  observation  is 
difficult  to  evaluate  in  light  of  the  large  difference  in  sample  sices* 
However*  it  is  unlikely  that  a  larger  sample  of  320  disease~free  males 
would  result  in  a  scatter  diagram  similar  to  that  of  the  320  diseased 
males* 

To  obtain  a  nonparametric  density  contour  plot  we  computed  a 
bivariate  product  kernel  estimate  (Epanechnikov*  1969)  given  by 


using  a  quartic 


f(x,y)  * 


_ L_  | 

nhxhy  i*l 


(biweight)  kernel 


(1) 


R(x)  *  jf<l-*2)2  ![-!.!](*)  <2) 

and  preliminary  values  of  the  smoothing  parameters  given  by 
hx  c  2  n  1 ^  where  a^  represents  a  trimmed  and  pooled  estimate  of  the 
standard  deviation  for  the  two  groups  with  a  similar  expression  for  hy. 
Density  values  were  computed  over  a  grid  of  150  by  90  points.  Whan 
applied  to  the  data  for  the  diseased  males*  the  contour  plot  reveals  a 
striking  bimodal  feature*  as  shown  in  Figure  2.  The  contours  of  equal 
probability  are  at  the  ten  levels  0.05  to  0.95  in  increments  of  0.10  as 
a  fraction  of  the  respective  maximal  modal  levels.  The  density  function 
of  the  disease-free  males  could  be  well  approximated  by  a  bivariate  Nor' 
mal  form.  Its  mode  coincides  with  the  left  of  the  two  modes  in  the  den- 
sity  function  of  the  diseased  males. 


The  contour  plots  have  helped  emphasice  a  feature  in  the  scatter 
diagram  that  might  have  gone  unnoticed.  The  contour  plots  also  aid  in 
compensating  for  the  difference  in  sample  sices.  The  discovery  of  the 
bimodal  feature  led  to  formulation  of  a  complex  cholesterol-triglyceride 
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interaction  in  the  model  for  estimating  the  risk  of  coronary  artery 
disease.  Clinically*  the  difference  of  50  mg/X  between  the  two  modes  in 
Figure  2  for  the  diseased  males  is  greater  than  the  reduction  in 
cholesterol  by  dietary  intervention  (which  usually  achieves  proportional 
reductions  in  the  range  of  10  to  15  percent). 

1.  IrivexUtl  Data 

The  data  presented  in  this  section  were  obtained  by  processing 
four-channel  Lendeat  data  measured  over  North  Dakota  during  the  summer 
growing  season  of  l?77  and  were  furnished  by  Dick  Heydorn  of 
NASA/Houston  and  Chuck  Sorensen  of  Lockheed /Houston.  The  sample  con¬ 
tains  approximately  21*000  points*  each  representing  a  1.1  acre  pixel* 
covering  a  5  by  6  nautical  mile  section.  On  each  pass  over  an  indivi¬ 
dual  pixel  by  the  Landsat  satellite*  the  four  channel  readings  were  com¬ 
bined  into  a  single  value  that  measures  the  "greenneas"  of  the  pixel  at 
that  time.  The  greenness  of  a  pixel  was  plotted  as  a  function  of  time 
from  the  five  paaaea  during  the  growing  season.  Finally*  Badhwar's 
(1982)  growth  model  was  fitted  to  this  curve.  This  modal  has  three 
parameters  which  are  contained  in  each  trivariate  data  point.  Tha  first 
variable  (x)  gives  the  time  the  "crop"  (if  any)  ripenad.  The  second 

I 

variable  (yj  measures  the  approximate  time  to  ripen.  And  the  third 
variably  (s)  measures  tha  level  of  "greenness"  at  the  time  of  ripening. 
Although  it  is  natural  to  group  these  data  by  actual  type  of  ground 
cover  for  classification  procedures*  we  have  not  done  so  here. 

It  is  not  possible  to  present  a  satisfactory  piccure  of  a  three- 
dimensional  scatter  diagram  of  these  data  for  this  article.  However*  on 
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an  ABD512  terminal  with  512  by  512  resolution*  a  projection  of  these 
data  onto  the  screen  typically  displayed  only  4000  points*  the  rest 
being  "hidden"  behind  displayed  points.  Viewed  from  several  different 
angles*  various  shapes  and  features  in  the  data  were  easily  perceived. 
Color  was  used  to  indicate  the  level  of  the  variable  perpendicular  to 
the  screen. 

We  can  present  density  contours  of  an  estimate  f(x»y»s).  Consider 
an  equiprobabla  contour  at  level  ct  that  is*  consider  those  points 
(x*y*s)  satisfying  the  aquation  f(x*y*s)  ■  c.  The  solution  of  this 
equation  for  a  smooth  density  estimate  f  is  a  smooth  surface  (or  sur- 
faces)  in  &  .  This  surface  may  ba  displayed  by  intersecting  it  with  a 
series  of  planes  displaced  equal  distances  along  the  co-ordinate  axes* 
in  the  following*  slong  only  the  x  and  y  axes.  In  Figure  3*  we  display 
the  surface  for  c  *  IX  of  the  maximal  mode  value*  Comparing  Figure  3  to 
the  corresponding  scatter  diagram  on  the  same  projection  plane  reveals 
how  surprisingly  little  of  the  data  space  is  enclosed  in  this  contour. 
In  the  scatter  diagram  our  eyes  focused  on  rays  of  points  that  seemed 
interesting  but  represented  only  a  small  fraction  of  the  data.  Also 
notable  in  Figure  3  is  a  cylindrical  shape  disjoint  and  behind  the 
larger  surface.  This  feature  was  also  clearly  visible  in  the  scatter 
diagram  and  represents  acres  in  which  sugar  beets  were  grown. 
Apparently  the  method  by  which  sugar  beets  are  harvested  leads  to  a 
singularity  in  the  estimation  of  the  growth  model  parameters  with  yCO. 

Expanding  the  scale  by  a  factor  of  2  while  retaining  the  same 
center  as  in  the  c  ■  IX  picture*  we  show  the  contour  shapes  at  levels 
c  »  10X,  30X*  and  50X  of  modal  height.  Notice  how  each  contour  shape 
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"fits"  inside  the  preceding  one*  Alao  observe  how  multimodal  features 
appear  in  this  space*  Three  modes  are  shown  in  this  sequence.  On  a 
color  graphics  terminal*  we  may  simultaneously  view  these  and  other  con¬ 
tours  by  using  different  colors  to  draw  each  contour. 

Again*  the  density  plots  have  complemented  and  added  to  our  under¬ 
standing  of  these  data*  It  is  easier  to  see  inside  the  data  cloud  with 
this  representation  and  also  makes  rotation  of  the  data  cloud  less 
important* 

A.  Computational  Conaidarationa 

A  new  algorithm  and  density  estimator  were  developad  to  display  the 
trivariate  contour  plots  and  we  hope  to  report  on  it  in  another  paper 
(Scott*  1983b).  8peed  is  an  important  factor  in  an  interactive  environ¬ 
ment.  The  kernel  method  used  in  the  bivariate  case  becomes  excruciat¬ 
ingly  alow  when  presented  with  21*000  points  in  three  dimensions.  In 
real  time*  a  few  minutes  were  required  on  a  Vax  11/780  to  compute  the 
bivariate  kernel  contours  for  320  points  on  a  130  by  90  mash.  To  gen¬ 
erate  the  pictures  in  Figure  3*  we  evaluated  the  density  on  a  30  by  30 
by  30  mesh  for  21*000  points.  A  straightforward  kernel  estimator  would 
have  required  several  hours  to  compute  I 

The  histogram  estimator  is  extremely  efficient  computationally*  but 
very  inefficient  statistically  --  and  relatively  more  inefficient  in 
higher  dimensions  than  kernel  methods.  One  recent  discovery  indicates 
that'  the  frequency  polygon  may  be  a  good  choice  of  a  nonparametric  den¬ 
sity  estimator  since  it  is  computationally  equivalent  to  a  histogram  but 
statistically  similar  to  a  kernel  estimate  (Scott*  1983a).  However*  the 
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frequency  polygon  in  several  dimensions  suffers  from  sensitivity  to 
choice  of  cell  boundaries.  The  new  algorithm  addresses  this  problem  and 
is  asymptotically  equivalent  to  a  certain  kernel  estimate.  Other  fast 
preliminary  estimates  in  one  and  two  dimensions  may  be  obtained  by 
numerical  approximation  of  kernel  estimates  in  place  of  statistical 
approximation*  which  we  prefer. 

i.  Where  Q&  fr  fin? 

We  do  not  really  know  for  how  many  dimensions  nonparametric  density 
estimates  will  be  useful  and  feasible*  8catter  diagrams  have  been  used 
in  a  highly  interactive  environment  to  visualise  nine-dimensional  data 
(Tukey*  Friedman*  and  Fisherkeller*  1976).  Many  possible  strategies  may 
be  envisioned  for  using  color  and  motion  to  examine  data  in  more  than 
three  dimensions.  We  expect  much  progress  in  this  area.  But  for  larger 
and  larger  data  eats  requiring  sophisticated  analysis*  we  believe  that 
density-based  methods  will  be  both  efficient  and  effective. 
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LOGISTIC  SUPPORTABILITY 


One  afternoon  of  the  28th  Conference  on  the  Design  of  Experiments 
In  Army  Research  Development  and  Testing  was  devoted  to  the 
Important  area  of  Logistic  Supportabillty.  First  on  the  agenda 
was  the  presentation  by  OTEA.  It  was  entitled  "Improving  the  Test 
and  Evaluation  of  Integrated  Logistics  Support  In  OT,"  and  Is 
published  In  these  proceedings  In  the  format  of  a  slide  presentation 
The  DARCOM  presentation  came  next  and  carried  the  title 
"Supportabillty  -  Requirements,  Design  Test  and  Evaluation." 
Unfortunately,  no  printed  matter  was  submitted  for  publication 
concerning  this  address.  The  final  presentation,  "Logistic 
Supportabillty  Testing  and  Evaluation  During  OT,"  was  given  by 
TRADOC  personnel.  Their  report  directly  follows  that  made  by  OTEA. 
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SITUATION 


THE  SOURCE  AND  FUNDING  FOR  THIS  INCREASE  IN 
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ABSTRACT :  ^Army  policy  requires  testing  and  evaluation  of  the  logistics 
supportabllity  of  a  system  in  the  acquisition  process.  The  thrust  Is  to  assess 
both  the  adequacy  of  system  design  for  support  and  the  adequacy  of  the  support 
package  developed  with  the  system.  While  the  policy  is  generally  sound,  the  lack 
of  a  comprehensive  Army  methodology  for  test  and  evaluation  of  logistics  support- 
ability  during  operational  testing  has  wsaksnad  implementation  of  the  policy. 

Presented  first  is  a  review  of  the  current  statue  of  operational  test  and 
evaluation  of  logistics  supportabllity.  The  key  ingredients  to  effective  and 
comprehensive  logistics  supportabllity  test  and  evaluation  are  identified.  For  each 
ingrediunt,  problem  areas  with  proposed  solutions  ara  discussed,  thus  providing  a 
TRADOC  perspective  of  where  we  are  and  where  we  should  be. 

Following  thu  roviow  is  TKADOC  proposed  methodology  for  oporatlonnl  testing  and 
evaluation  of  logistics  supportabllity.  All  phases  of  operational  testing  are 
addressed  with  incorporation  of  proposed  solutions  for  problem  areas. 


LOGISTICS  SU PORTABILITY  (LOGS) 

OPERATIONAL  TEST  AND  EVALUATION  (OT&E) 

METHODOLOGY 

1.  PURPOSE.  This  paper  provides  a  discussion  of  key  ingredients,  current 
problem  a^eas  and  proposed  solutions  for  accomplishment  of  effective  LOGS  test 
and  evaluation  during  operational  testing  (OT).  For  the  purposes  of  this 
paper,  LOGS  is  defined  as: 

Logistics  Supportability  (LOGS).  The  characteristics  of  the  system 
(materiel  and  crew)  and  the  related  support  elements  (support  ooncept,  support 
materiel,  and  support  personnel)  as  they  contribute  to  the  retention  and 
restoration  of  the  materiel  system  in  an  operational  effective  status. 
Therefore,  LOGS  is  the  way  these  three  elements  affeot  and  are  affected  by  the 
materiel  system. 

2.  Comprehensive  effective  OT&E  of  the  supportability  of  a  system  can  be 
divided  into  three  categories:  the  decision  prooess,  analysis  and  resources. 
Within  each  of  these  categories,  there  are  key  ingredients  to  an  effective 
logistics  evaluation.  Below  is  a  disoussion  whioh  provides  a  list  of  dileranas 
with  proposed  solutions  for  eaoh  key  ingredient.  Proposed  LOGS  operational 
test  and  evaluation  methodology  providing  specifio  guidanoe  is  at  Annex  A.  It 
is  assumed  that  waivers  will  not  be  granted  to  any  of  the  key  ingredients. 

When  a  waiver  is  granted,  it  becomes  a  dilenma  in  itself. 

a.  Decision  process. 

(1)  Recognition  of  LOGS  as  a  mission  oriented  issue. 

DILEMMA  -  Decision  makers  often  perceive  operational  effectiveness 
and  combat  power  issues  as  having  higher  visibility  and  being  the  most  mission 
oriented,  therefore,  are  highest  priority.  Thus,  in  the  battle  for  resources, 
LOGS  issues  are  often  sacrificed,  since  supportability  issues  which  require 


f 

J 

I 

g-eater  time  and  manpower  than  effectiveness  cr  combat  powe1". 

;  SOLUTION  -  All  LOGS  issues  must  be  critically  reviewed  by  the 

|  decision  maker  at  each  decision  point  with  data  provided  showing  LOGS  inpact 

on  combat  power. 

(2)  Acquisition  schedule  to  accommodate  effective  LOGS  OT&E. 

j  DILEXMA  -  The  acquisition  cycle  is  decision  milestone  driven. 

Therefore  with  OT  following  developmental  tests  (DT),  as  slips  occur  in  DT, 
the  time  available  for  OT  decreases.  Thus  effective  LOGS  OT&E  is  compromised . 
^  SOLUTION  -  Existing  policies  need  to  be  enforced.  Additional 

policies  needed  are  as  follows: 

(a)  When  slip  occurs  in  critical  program  milestones  (e.g.  DT  start 
or  complete),  subsequent  milestones  will  be  slipped  accordingly. 

(b)  When  critical  LOGS  elements  (e.g.,  system  peculiar  Test 

(  Measurement  Diagnostic  Equipment  (TWDE)  and  generators,  DS  and  GS  support 

i 

package,  etc.)  will  not  be  tested  in  OT  XX  of  the  system,  OT  II  testing  of 
these  support  items  will  be  accomplished  prior  to  their  type  classification. 

I  This  currently  happens  for  training  devices  since  they  have  a  separate 

requirement  document.  However,  other  items  encompassed  by  the  system 
requirement  document  do  not  undergo  this  necessary  testing. 

(3)  Test  and  logistics  test  resources  waivers  held  to  a  minimum. 

DILEMMA  -  Current  waiver  procedures  are  confusing.  Policy  (AK  71-3, 

AR  700-127,  etc.)  provides  a  myriad  of  waiver  approval  procedures  and 
authorities  dependent  on  resource,  test,  and  system  category  of  concern  and 
situation.  Procedures  and  guidelines  are  not  clear  or  consistent  for  all 
situations. 

SOLUTION  -  Have  only  one  procedure  for  waiving  tests  and  test 
resources  with  well  defined  procedures  and  guidelines. 
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(1)  LOGS  OT&E  methodology  understood  and  consistently  applied  across  the 
Army  community. 

DILEMMA  -  There  i '  no  DA  published  baseline  methodology  on  LOGS  OT&E 
to  guide  and  harmonize  the  comaunity.  Recently  drafted  DA  pamphlet  on 
supportability  test  and  evaluation  provides  broad  OT  guidance  but  refers  to  AR 
71-3  and  DA  Pam  71-3  for  specifics.  These  specifics  have  not  been 
incorporated.  TRADCC  developed  and  implemented  a  detailed  methodology  in 
coordination  with  all  the  Army  comnunity  in  1977.  However,  this  methodology 
has  not  been  since  updated  Or  sanctioned  for  all  to  use  by  DA, 

SOLUTION  -  A  methodology  be  developed  and  incorporated  in  a  DA 

panphlet . 

(2)  Critical  supportability  issues  and  criteria  clearly  and  appropriately 
defined. 

-  <*•- 

DILEMMA  -  Issues  and  criteria  have  not  effectively  encompassed 
operational  readiness  of  system  and  logistics  burden  emplaced  by  system. 

|  SOLUTIONS: 

!  (a)  Set  three  critical  LOGS  issues,  (1)  operational  readiness,  (2) 

logistics  burden,  and  (3)  “system  support  package  deficiencies. 

| 

•System  support  package  (SSP)  -  A  composite  package  of  support  elements  in 

initial  issue  quantities  for  a  materiel  system  in  the  operational  (deployed) 

environment.  For  OT  it  is  a  combination  of  test  support  packages  provided  by 

I 

.  materiel,  combat  and  training  developers.  At  OT  II  content  is  scaled 

commensurate  with  the  force  slice  being  played  to  evaluate  organizational, 
direct  support  and  general  support  capabilities. 
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(b)  Insure  requirements  documents  provide  a  sound  basis  for  defining 


critical  criteria. 

(3)  Appropriate  evaluation  techniques  defined  and  available. 

DILEMMA  -  Current  methodologies  in  TRADQC  or  DARCOM  have  not  provided 
analytical  techniques  for  establishing  logistics  burden  criteria  and 
evaluating  operational  readiness  or  logistics  burden  achievement. 

SOLUTIONS: 

(a)  Provide  modeling  and  analysis  routines  for  developing  oriteria 
for  logistics  burden.  Recently  published  TRADOC/DARCOM  Pam  71-11  provides 
techniques  for  developing  system  readiness  objective  (SRO)  requirements  which 
also  apply  to  criteria. 

(b)  Provide  modeling  and  analytical  routines  for  evaluating 
achievement  of  readiness  measures  and  logistios  burden. 

(c)  Computerize  modeling  and  analytical  routines  and  make  available 
to  all  OT  issue-criteria  developers,  testers  and  evaluators. 

c.  Resources . 


(1)  Operator,  orew,  maintenance  and  supply  personnel  in  the  appropriate 
specialty  and  trained  as  planned  when  fielded. 

DILEMMA  -  Same  as  addressed  for  decision  process  key  ingredients. 
SOLUTION  -  Same  as  addressed  for  decision  process  key  ingredients. 
Conduct  of  OT  I  is  critical  to  determining  personnel  skill  and  training 
requirements  for  development  during  following  phase. 
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1 

(2)  System  and  support  gear  (hardware  and  ^software)  suitably  mature  for 

test.  j 

DILEMMA  -  Same  as  addressed  for  decision  process  key  ingredients. 

SOLUTION  -  Sane  as  addressed  for  decision  process  key  ingredients. 

Conduct  of  OT  I  is  critical  to  identifying  materiel  system  and  support  gear  j 

defioiencies  for  improvement  during  next  phase  so  that  sufficiently  mature 
items  ere  provided  for  OT  II. 

(3)  Sufficient  sxice  of  the  force  to  place  proper  logistics  demands  on  j 

the  support  structure. 

DILEMMA  -  OT  test  players  and  data  collectors  are  provided  from 

I 

active  FORSCOM  units  which  have  other  missions  thus  constraining  their  j 

availability  for  test. 

SOLUTIONS: 

(a)  The  evaluator  must  utilize  to  the  maximun  other  data  souroes. 

(b)  Work  with  DT  testers  to  incorporate  representative  trained 
troops  in  their  test  as  maintenance  personnel  or  perhaps  have  DT  off  system 
repairables  shipped  to  the  OT  site  for  maintenance. 

(c)  Work  with  materiel  developers  to  have  representative  trained 
troops  used  in  their  logistics  demonstrations  and  PTEAR. 

(d)  Require  both  OT  II  and  FOE  be  performed  on  a  routine  basis  and 
combine  data  sets  to  degree  possible  for  a  more  comprehensive  evaluation  prior 
to  fielding. 

^Support  software  -  Includes  entire  set  of  programs,  procedures,  and  related 
docunentation  such  as  technical  manuals  and  computer  programs  necessary  for 
supply,  maintenance  and  training. 
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(e)  Evaluation  models  addressed  above  must  provide  for  expansion 
beyond  the  unit  and  support  structure  sizes  tested  to  provide  a  -epresentative 
slice  of  force  structure. 

(f)  Consider  combining  several  small  scale  OT  occurring  in  same  year 
into  a  single  larger  scale  test. 

(g)  Explore  use  of  DA  sample  data  collection  program,  National 
Training  Center  (NTC)  and  field  exercies  (e.g.,  REFORGER)  to  expand 
supportability  data  base  during  initial  fielding. 

(4)  Sufficient  test  time  to  properly  exercise  the  critical  support 
elements . 

DILEMMA  -  While  our  test  guidelines  use  RAM  as  a. standard  for 
determining  the  test  length,  there  is  no  assurance  that  all  critical  LOGS 
elements  will  be  functioned,  including  both  "primary  and  ""secondary  logistics 
systems . 

SOLUTIONS: 

(a)  Have  simulated  maintenance  during  OT  such  that  a  suitable 
percentage  of  oritioal  tasks  are  acocmplished  at  eaoh  maintenance  level 
through  GS. 

(b)  Conduct  sample  data  collection  programs  on  a  routine  basis  with 
initial  fielding  and  provide  results  to  OT  evaluators  and  testers. 

"Primary  support  system  -  Personnel,  maintenance  and  supply  system  which 
responds  directly  to  the  materiel  systsn  under  consideration. 

""Secondary  support  system  -  Maintenance  and  supply  system  which  responds  to 
the  primary  support  system. 


(c)  Same  as  (f)  for  key  Ingredient  concerning  force  slice. 

(5)  Adequately  defined  and  implemented  tactical  scenario  and  support 
concept  for  test. 

DILEMMA  -  Test  support  packages  provided  by  materiel,  combat  and 
training  developers  for  OT  are  often  incomplete  or  do  not  properly  define 
soenario  and  support  ooncepts. 

SOLUTIONS! 

(a)  Improved  quality  control  of  support  packages. 

(b)  Improved  waiver  approval  procedures  as  described  above  for  the 
decision  process  key  element. 

(c)  Readiness  for  test  reviews  conducted  and  results  brought  to 
decision  makers  attention. 

(6)  Effective  data  collection  system  to  satisfy  issues  and  orlterla. 

DILEMMA  -  Data  collection  system  is  fragnented  due  to  laok  ol 

standardization,  coordination,  and  communication  aoross  the  Array  OT  community 
resulting  in  duplicated  efforts. 

SOLUTIONS; 

(a)  Use  all  data  sources. 

(b)  Simulated  maintenance  actions  during  OT. 

(c)  Develop  a  standard  set  of  programs  for  OT  to  assess  repair  parts 
and  POL  consumption,  stock  number  analysis  of  tools,  7MDE,  and  special 
equipment,  assess  off-line  recoverable  repairables  and  training,  and 
validating  and  verifying  manuals,  and  maintenance  allocation  charts.  ADP 
resources  should  be  used  to  the  maximum  degree  for  this  methodology. 

(d)  Obtain  from  Soldier  Support  Center  the  profile  of  the 
qualifications  for  MOS  test  player  personnel.  This  is  very  critical  when  a 
small  personnel  sample  is  being  used  since  it  is  possible  to  obtain  people  not 


in  the  profile  but  having  the  MOS. 

(e)  Data  collectors  should  be  carefully  chosen  and  trained  with 
specific  skills  and  therefore,  they  would  not  solely  perform  the  function  of 
data  collection  but  be  a  system  evaluator. 

(7)  Personnel  adept  and  qualified  in  logistics  supportability  on  both  the 
test  and  evaluation  teams  as  well  as  in  those  agenoies  responsible  for 
defining  the  issues  and  oriteria. 

DILEMMA  -  Probably  the  most  serious  defiolenoy  in  LOGS  testing  and 
evaluation  is  the  lack  of  trained  logistlos  testers  and  evaluators.  Not  only 
are  individuals  assigned  to  a  testing  or  evaluation  agenoy  without  training  in 
techniques  of  testing  or  evaluation,  but  they  are  not  trained  in  the  specific 
area  of  logistics  supportability. 

SOLUTION  -  Match  SC  personnel  with  the  system  to  be  tested  and  train 
them  in  LOGS  management  techniques,  LOGS  quantitative  teohniques,  and 
logistics  support  analysis.  These  courses  are  readily  available  and  should  be 
mandatory  for  all  logistics  testers  and  evaluators  prior  to  assignment. 
Likewise,  civilian  logistlos  test  and  evaluation  personnel  need  similar 
training . 

3.  CONCLUSION. 


In  conclusion,  the  current  regulation  guidelines  provide  a  system  fbr 
addressing  LOGS  Issues;  however,  total  adequacy  of  LOGS  evaluation  is  resouroe 
dependent  and  trade-offs  generally  reduoe  the  credibility  of  operational  test 
results  for  logistics  support,  and  unless  these  problems  are  solved,  the 
perception  of  inadequate  LOGS  testing  and  evaluation  will  persist. 
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TEST  AND  EVALUATION 
LOGISTICS  SUPPORTABILITY  (LOGS)  IN  OT 


1.  INTRODUCTION.  The  test  and  evaluation  of  logistics  supportability  is 
addressed  in  numerous  test  related  documents  within  the  US  Army.  One  of  the 
dilemmas  main  addressed  in  tne  paper  is  that  there  is  no  DA  operational  test 
and  evaluation  procedure  documented  and/or  implemented  to  assure  that 
logistlos  is  comprehensively  evaluated.  The  methodology  proposed  herein  to 
fill  this  Dept  of  the  Army  void  is  a  refinement  of  the  methodology  developed 
by  and  presently  being  used  within  TRADOC.  It  is  assumed  that  the  remaining 
dilemmas  discussed  in  the  main  paper  have  been  resolved. 

2.  DEFINITIONS.  For  the  purposes  of  this  methodology,  the  following 
definitions  apply: 

a.  Log 1st lo s  Supportability  (LOGS).  The  characteristics  of  the  system 
(materiel  and  orew)  and  the  related  support  elements  (support  concept,  support 
materiel,  and  support  personnel)  as  they  contribute  to  the  retention  and 
restoration  of  the  materiel  system  in  an  operational  effeotlve  status. 
Therefore,  LOGS  is  the  way  the  three  elements,  support  conoept,  support 
materiel  and  support  personnel,  affeot  and  are  affected  by  the  materiel 
system. 

b.  System  Support  Package  (SSP),  A  composite  package  of  support  elements 
in  initial  issue  quantities  planned  for  a  materiel  system  in  the  operational 
(deployed)  environment.  For  OT  it  is  a  combination  of  test  support  packages 
provided  by  materiel,  combat  and  training  developers.  In  its  preliminary 
form,  it  is  provided  before  and  evaluated  during  developmental  and  operational 
testing  and  evaluation  o  validate  the  organizational,  direct  support,  and 


general  support  maintenance  capabilities.  For  logistic  supportability 
testing,  it  normally  includes: 


I 

(1)  Support  and  test  equipment. 

(2)  Trained  personnels  (including  the  training  programs,  materials, 
devioes,  and  anmunition  needed  to  develop  those  skills). 

(3)  Supply  support.  ! 

(4)  Technical  logistio  data. 

(5)  Facilities. 

(6)  Computer  resources. 

(7)  Maintenance  support. 

(8)  The  logistics  concept. 

c.  Primary  Support  System.  Personnel,  maintenance  and  supply  system 
which  responds  directly  to  the  materiel  system. 

d.  Secondary  Support  System.  Maintenance  and  supply  system  which 
responds  to  the  primary  support  system. 

e.  Loglstlos  Support  Conoept.  The  overall  "how*'  the  logistics  system  is 
set  up  and  administered  to  support  the  materiel  system.  The  support  oonoepts 
provide  the  organizational  structure  and  responsibilities  for  accomplishing 
the  maintenance  and  supply  functions  ut  eaoh  level.  This  inoludes  the 
identification  and  allocation  of  hardware,  software  and  support  personnel  to 
eaoh  supply  and  maintenance  level. 

f .  Logistics  Support  Materiel.  Those  hardware  and  software  items  needed 
for  supply,  maintenance  and  training  support.  Logistics  support  hardware 
includes,  test  measurement  and  diagnostic  equipment,  special  and  ccnmon  tools, 
repair  parts,  resupply  and  rearm  vehicles,  training  devioes  and  similar 
equipment  assigned  to  supply,  maintenance  and  training  units.  Logistics 
support  software  includes  the  entire  set  of  programs,  procedures,  and  related 
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doc mentation  such  as  technical  manuals,  lubrication  orders,  computer 
programs,  etc.,  necessary  for  maintenance,  supply  and  training. 

g.  Logistics  Support  Personnel.  Selection  criteria  and  training  required 
for  operator,  orew,  maintenance  and  supply  personnel.  Personnel  selection 
entails  definition  of  duty  requirements  and  the  skills  and  characteristics 
needed  to  retain  the  system  in  or  then  failure  occurs,  restore  it  to  an 
operationally  effective  condition.  The  amount  and  type  of  training  is  a 
funotion  of  the  system  complexity  and  the  designated  military  oooupation 
specialty  (MOS) . 

h.  Materiel  System.  Mission  item  being  acquired  for  whioh  LOGS  is  to  be 
evaluated.  Characteristics  include  all  faotors  of  design  which  affeot 
logistics  support.  Examples  are,  design  for  maintainability,  human  faotors 
and  safety  affect  the  effioienoy  and  speed  of  maintenance  operations. 

Hardware  requirements  for  special  handling,  training  devloes,  frequency  of 
calibration,  IMEE,  transportability,  resupply  equipment  affeot  end  item 
logistios.  Design  for  standardization  within  the  Army  and  rationalization, 
standardization  and  interoperability  (RSI)  with  other  servloes  and  NATO  is 
also  considered. 

3*  EVALUATION  PROCESS.  To  assure  that  a  materiel  system  in  the  aoquinition 
oyole  oan  be  fully  supported  when  fielded,  logistios  supportablllty 
assessments  should  be  conducted  during  every  phase  of  the  acquisition  cycle. 
Geniune  logistios  supportablllty  assessments  do  not  just  happen  as  a  result  of 
operational  testing.  Evaluation  planning  Is  key  to  timely  and  effective 
operational  LOGS  assessment  as  with  any  other  system  performance  parameter. 

The  first  step  in  the  evaluation  process  is  the  identi float ion  of  issues  with 
associated  criteria  which  must  be  addressed  by  the  decisioin  milestone  of  the 


acquisition  cycle.  The  combat  developer  or  training  developer  provides  the 
issues  and  criteria  to  the  evaluator.  Subsequently,  the  evaluator  must 
identify,  for  a  given  program,  the  evaluation  approach  or  methodology,  the 
analysis  to  be  performed ,  the  data  required  and  the  sources  of  the  data.  This 
evaluation  planning  information  is  documented  in  the  independent  evaluation 
plan  (IEP).  Required  operational  tests  (OT)  are  performed  ly  the  tester. 

Onoe  the  required  data  becomes  available,  the  evaluator  performs  the 
prescribed  analyses  to  evaluate  each  issue  and  develop  his  overall  assessment 
of  the  system  and  testing  done  to  date.  Hence,  evaluation  of  LOGS  involves 
the  following  steps: 

a.  Identification' of  critical  issues  and  criteria. 

b.  Evaluation  planning. 

c.  Perform  studies/ conduct  testing/ collect  data. 

d.  Evaluation. 

4.  ISSUES  AND  CRITERIA.  The  first  step  in  effective  LOGS  evaluation  is 
identification  of  issues  and  criteria  whioh  must  be  addressed  and  data 
provided  for  an  adequate  resolution  by  time  of  the  decision  milestone.  There 
are  three  critical  supportabilitiy  issues  to  which  decision  makers  need 
answers  before  making  the  production  decision  as  follows: 

a.  Does  the  system,  when  supported  in  accordance  with  the  approved 
logistics  concept,  achieve  required  operational  readiness? 

This  issue  examines  both  the  design  of  the  system  for  effective 
support  and  the  ability  of  the  support  system  to  rapidly  respond  to  the  system 
need  for  maintenance.  Operational  availability  or  other  system  readiness 
objective  (SRO)  measure  will  be  the  principal  criteria  for  this  issue. 
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TRADOC/DARCOM  Para  71-11  provides  guidance  on  development  of  these  values  and 
also  may  be  used  for  guidance  in  evaluating  the  SRO. 

b.  Does  the  system  impose  excessive  burden  on  any  of  its  support 
elements? 


This  issue  examines  demands  placed  on  oritioal  support  elements  and 
resouroes  required  and  available  to  meet  these  needs.  Unit  quantity  rather 
than  single  system  support  burden  is  of  dominant  concern.  In  those  cases 
where  the  SRO  (issue  "a”  above)  is  not  achieved,  two  levels  of  evaluation  will 
be  made.  Oie  is  the  resouroes  required  for  the  achieved  SRO.  The  other  i3 
estimated  additional  resduroes  neoessary  to  achieve  the  required  SRO. 

Criteria  will  include  measures  such  as:  2  MOS  XX  man-years  at  organizational 
level  and  2000  gal  MOGAS  per  day. 

c.  Are  there  any  deficiencies  in  the  system  support  package? 

This  issue  examines  the  completeness,  appropriateness,  aocuraoy,  and 
adequacy  of  logistic  elements  in  the  test  support  packages  provided  for  OT. 
This  issues  is  subdivided  into  four  key  areas  which  enoompass  the  AR  700-127 
Integrated  Logistics  Support  elements  of  conoem.  These  four  areas  are 
logistics  concept,  support  materiel,  support  personnel  and  materiel  system 
characteristics.  Criteria  for  this  issue  ars  generally  subjective  requiring 
Judgment  or  direct  observation  by  testers.  For  exanple,  maintenance  tasks  and 
resources  allocated  at  proper  level. 

d.  Issue  and  criteria  dendritic.  A  complete  dendritic  of  the  three 
issues  is  provided  at  Appendix  A, 


5.  EVALUATION  PUNNING. 


a.  Material  Acquisition  Phases.  To  assure  that  a  materiel  system  in  the 
acquisition  process  can  be  effectively  supported  when  fielded,  supportability 
assessments  occur  during  every  phase  of  the  acquisition  process.  However, 

OT&E  evaluations  occur  in  three  phases  with  emphasis  as  follows: 

(1)  Demonstration  and  Validation  Phase.  DT  I  and  OT  I  are  conducted 
during  this  phase  using  advanced  development,  breadboard  or  brassboard 
prototypes.  Generally,  contractor  or  developer  taining  is  provided  and 
concentration  is  on  the  developmental  system.  Early  doctrine,  organization 
and  logistic  ooncepts  are  available.  Generally,  there  is  competition  between 
alternative  system  oonoepts  to  continue  to  next  phase.  The  evaluation  focus  is 
on: 

(a)  Analyses  of  supportability  merits  of  the  competing  system. 

(b)  Identify  system  and  support  ooncept  modification  needed. 

(o)  Identify  special  requirements  for  personnel,  TMDE,  and  training  to 
support  development  in  the  next  phase. 

(d)  Refine  OT  I  Issues  and  criteria  and  identify  any  new  ones  for  OT  II. 

(2)  Full  Soale  Engineering  Development  Phase.  This  is  the  phase  leading 
to  full  production  deoision  with  that  rare  exception  where  Low  Rate  Initial 
Production  (LRIP)  is  authorized.  DT  II  and  OT  II  tests  ooour.  Answers  to  the 
three  oritioal  issues  (readiness,  burden  and  system  support  package)  are 
needed. 

(3)  Production  and  Deployment  Phase.  Evaluation  oonduoted  in  this  phase 
concentrates  on  answering  unanswered  or  unresolved  issues  and  verification  of 
support  deficiencies  frcm  the  previous  phase. 

b.  Analytical  methods.  For  each  phase  of  the  materiel  acquisition 
process,  once  the  LOGS  issues  and  criteria  have  been  finalized,  the  next  step 


is  to  establish  the  evaluation  scheme  to  ascertain  the  analyses  techniques 
be  utilized  and  and  identify  the  data  requirements  and  sources. 

(1)  Operational  Readiness.  Readiness  is  a  function  of  system 
utilization,  maintenance  requirements  and  administrative  and  logistics 
downtime.  Normally,  the  readiness  parameter  for  Army  systems  will  be 
operational  availability.  The  equation  for  operational  availability  is  as 
follows : 


OT  +  ST 

Ao  = _ 

OT  +  ST  +  TCM  +  TPM  ♦  TALDT 

Where: 

OT  *  operating  time  during  a  given  calendar  time  period. 

ST  s  Standby  time  (not  operating,  but  assumed  operable)  during  that 

period. 

TCM  a  Total  oorreotive  maintenance  downtime  in  clock  hours  during  that 

period. 

TPM  *  Total  preventive  maintenance  downtime  in  clock  hours  during  the 
OT  period. 

TALDT  s  Total  administrative  and  logistics  downtime  spent  waiting  for 
parts,  maintenance  personnel,  or  transportation  during  the  time  stated  period. 
Furthermore,  TALDT  is  a  function  of  (1)  operational  mission  reliability,  (2) 
percentage  of  operational  mission  failures  requiring  parts,  (3)  probability  of 
required  part  being  on  the  Prescribed  Load  list  and  Authorized  Stockage  List, 
(4)  probability  of  required  parts  being  in  stock  and  (5)  delay  times 
encountered  at  various  levels  of  maintenance.  Predicted  values  of  the  above 

I 

I  parameters  are  used  to  develop  the  Ao  value.  Once  the  system  is  sufficiently 

i 

i 
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matured  and  the  logistics  support  developed,  actual  system  data  may  replace 
predictions  to  give  a  better  quantitative  estimate  of  the  equipment  and  the 
support  system  to  achieve  the  SRQ.  Although  Ao  cannot  be  measured  directly  in 
an  operational  test,  testing  can  generate  data  on  the  maintenance  requirement, 
operational  mission  reliability,  percentage  of  operational  mission  failures 
requiring  parts,  and  the  probability  of  required  parts  being  on  the  stockage 
lists  at  various  levels  of  maintenance.  Substituting  this  data  and  utilizing 
the  remaining  data  used  in  developing  the  requirements ,  one  oould  develop  a 
more  valid  estimate  of  the  SRO.'  Sensitivity  analysis  could  indicate  needed 
improvements  such  as  the  pay  off  for  improving  logistics  support 
characteristics  of  the  system  or  refining  the  PLL  or  ASL. 

(2)  Logistics  burden.  The  purpose  of  these  analyses  is  to  determine  the 
strengths  and  weaknesses  of  planned  support.  Basically,  maintenance,  supply 
and  transportation  demands  plaoed  on  the  support  system  are  compared  against 
the  the  resources  provided  in  the  planned  support  system.  Consider  the 
examples  at  Appendix  B  as  follows 

(a)  In  the  manpower  analysis  example,  although  the  reliability 
requirement  is  nearly  met,  the  reliability  degradation  translates  into 
approximately  three  times  as  many  failures  as  projected.  Accordingly,  if  no 
reliability  improvement  is  obtained,  an  additional  43  maintenance  personnel 
will  be  required  to  support  the  system. 

(b)  In  the  fuel  consumption  analysis,  meeting  half  the  criteria  for  fuel 
consunption  translates  into  double  the  requirement  tankers,  bladders  and 
manpower  supporing  at  the  resupply  point  than  programed  for  the  system. 

Without  redesign  or  adding  these  needed  assets,  a  fifty  percent  reduction  in 
operational  readiness  (i.e.,  operational  availability)  will  result. 

(c)  The  TMDE  analysis  demonstrates  how  the  difference  between  foroe  slice 
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tested  and  to  be  supported  in  the  field  impacts  burden.  While  no  problem  with 
queing  at  the  1MDE  device  was  observed  in  test,  considering  the  force  slice  to 
be  fielded  shows  a  queing  problem  will  result.  The  analysis  also  suggests  how 
consideration  of  comunt  losses  can  impact  the  analysis  conclusions.  While  the 
first  analysis  concludes  operational  readiness  (i.e.,  operational 
availability)  will  be  significantly  degraded  because  of  the  queing  problem, 
considration  of  ocrobat  losses  may  result  in  determining  the  allocation  of  TMDE 
adequate.  Expected  systems  lost  in  the  first  three  days  of  battle  oould 
remove  the  queing  problem. 

(d)  At  present  there  is  no  documentation  or  regulatory  guidance  requiring 
a  specific  logistics  analysis  to  be  performed.  It  would  also  be  beneficial  to 
document  in  future  DA  pamphlets  several  examples  of  typical  maintenance, 
supply  and  transportation  burden  analyses.  However,  it  should  be  realized 
that  there  is  no  "cookbook"  method;  evaluations  and  analysis  techniques  must 
be  tailored  to  the  specific  system.  Analytical  techniques  should  be  kept 
simple  and  provide  timely  response.  Modeling  such  as  Maintenance  and 
Logistics  Analysis  (MALA)  analysis  is  too  complex  and  not  responsive. 

(3)  Support  Package  Deficiencies.  Test  support  packages  provided  by  the 
materiel,  combat  and  training  developers  define  the  support  struoture  and  its 
operation  and  provide  the  personnel,  materiel  and  software  upon  whioh  it 
functions.  Errors  and  inadequacies  are  generally  found  in  these  elements 
during  conduct  of  test.  Subjective  analysis  is  then  made  of  these  errors  and 
inadequacies  to  determine  their  level  of  severity  with  regard  to  functions  of 
the  support  system.  Further,  because  of  acquisition  program  constraints  or 
strategy,  elements  may  not  be  available  at  time  of  test.  These  must  be 
identified  and  assessed  as  to  impact  on  other  areas.  These  deficiencies  may 
or  may  not  have  impact  on  operational  readiness  or  logistics  burden  analysis 
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above.  For  example,  unnecessary  tools  would  not  affect  either,  where  as, 
inability  to  transport  in  C130  or  C141  aircraft  affects  burden  due  to 
conrcitment  status  of  C5A  aircraft.  The  analysis  should  result  in 
categorization  of  the  errors  and  inadequacies  as  follows: 

(a)  Significant  deficiency  -  makes  system  unacceptable  for  deployment  or 
correction  involves  more  than  the  most  routine  engineering.  Verification  of 
correction  needed  prior  to  full  production  decision. 

(b)  Other  deficiency  -  impacts  system  supportability  but  does  not 
constitute  a  significant  deficiency.  Verification  of  correction  required 
before  or  at  time  of  initial  fielding. 

(0)  Shortcoming  -  doesn't  significantly  impaot  system  supportability  but 
correction  should  be  made  if  possible.  Verification  of  correction  not 
required . 

0.  feta  Sources.  Once  the 'issues  and  criteria  have  been  defined  together 
with  analysis  techniques  for  their  evaluation,  the  next  step  is  to  identify 
data  sources  from  whioh  appropriate  data  oan  be  obtained  to  support  the 
evaluation  planned.  Potential  sources  include  but  are  not  limited  to  the 
following : 

(1) -  OT  will  be  the  dominant  data  source  for  the  independent  operational 
evaluation  of  supportability.  OT  is  conducted  with  representative  user 
operators,  crews  and  units  in  as  realistic  an  environment  as  possible. 
Operations  are  tactical  scenario  driven.  The  support  system  (personnel, 
equipment,  software,  procedures  and  organization)  is  as  olose  as  possible  to 
that  for  the  system  when  fielded.  Fbwever,  beoause  of  limitations,  other 
sources  should  be  investigated  to  determine  if'  they  provide  a  realistio  source 
for  data  not  achieveablo  in  the  OT. 

(2)  Consideration  should  be  given  to  the  possibility  of  combining  several 
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OT  scheduled  to  occur  in  the  same  year  into  a  single  large  scale  test.  This 
may  allow  for  a  longer  test  with  less  total  impact  on  FORSCOM  resources.  The 
larger  force  slice  for  test  would  provide  a  more  realistic  evaluation  of  the 
total  support  structure  when  fielding  occurs. 

(3)  DT,  while  a  technically  oriented  teat,  can  provide  data  of  benefit  to 
the  OT  evaluation.  Some  areas  include  transportability  analysis,  TMDE  and 
calibration  equipment  functional  accuracy  and  reliability,  RSI  compatibility, 
component  interchangeability,  and  technical  accuracy  of  documentation.  Data 
on  hardware  failure  frequenoy  which  places  demands  on  the  support  system  can 
also  be  provided.  However,  because  of  differences  in  technical  and 
operational  environments,  demands  may  be  signifioantly  different  from  the  OT. 
It  may  be  possible  to  bring  in  trained  representative  troops  to  perform 
maintenance  tasks  in  DT  whioh  because  of  test  duration  are  not  expected  to 
occur  in  OT.  Likewise  a  possibility  exists  to  have  DT  ship  off  system 
repairables  to  OT  for  repair.  Thus,  DT  oan  fill  an  OT  data  void  on  personnel 
seleotlon,  training,  manual  and  support  equipment  adequaoy,  human  faotors  and 
system  maintainability, for  those  tasks. 

(U)  FETE  are  user  tests  oonducted  to  address  issues  oonoerning  dootrine, 
organization  and  training.  Sane  are  oonducted  on  systems  during  the 
acquisition  process.  As  ouch,  these  may  be  appropriate  data  souroes  for 
issues  oonoerning  the  logistios  concept,  personnel  selection  or  training. 

When  conducted,  these  may  expand  the  OT  data  base. 

(5)  Logistioi?  Demonstration  (LD)  and  Preliminary  Teardown  Analysis 
(PTEAR)  oonducted  by  the  materiel  developer  may  also  be  a  valid  souroe  similar 
to  DT.  The  LD  is  a  special  experiment  to  address  technical  logistios  issues 
not  satisfied  by  other  tests.  The  PTEAR  provides  data  on  manual  aoouraoy  by 
actually  going  through  the  procedures  and  performing  the  various  maintenance 
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tasks  on  the  system. 

(6)  Skill  Performance  Aids  (SPA)  verification  is  conducted  by  the 
training  developer  on  those  systems  requiring  SPAS  manuals.  Readability  is 
determined  by  using  a  statistically  valid  sample  of  the  representative  MQS 
soldiers  to  use  the  manual.  For  other  manuals,  desk  audits  are  performed  by 
the  combat  and  training  developers.  Conducted  before  OT  II,  these  efforts 
serve  to  assure  manuals  received  are  more  mature  and  representative  of  that  to 
be  fielded.  Conducted  after  the  test,  they  may  serve  to  verify  correction  of 
manual  deficlences  found  in  test. 

(7)  Sample  Deta  Collection  Programs  are  conducted  by  DARCCM  commands 
responsible  for  readiness.  As  new  systems  are  fielded,  data  collection  teams 
monitor  the  system  and  oollect  specific  data  as  defined  in  a  data  collection 
plan.  These  programs  vary  in  length  from  six  months  to  in  exoess  of  one  year. 
These  are  good  data  sources  for  further  expansion  of  the  LOGS  data  base. 
Opportunity  to  submit  data  elements  and  other  involved  in  this  effort  is 
available  through  DARCOM. 

(8)  The  National  Thaining  Center  (NTC)  may  be  a  valid  data  source  for 
support ability  evaluation  during  early  fielding.  Data  colleoted  from  this 
source  would  provide  for  evaluation  of  supportability  under  varied  tactical 
soenario3.  It  should  be  noted  that  NTC  is  not  to  be  a  data  collection  agency. 
However,  it  may  be  possible  to  obtain  supportability  dat  they  routinely 
collect  or  to  find  work  around  solutions  through  coordination  with  NTC  staff. 

(9)  Field  exeroises  such  as  REFORGER  may  also  provide  valid  source  for 
supportability  data  during  initial  fielding.  This  source  should  be  explored 
to  determine  ability  to  provide  suitable  supportability  data  either  through 
their  routine  data  collection  or  special  collection  effort  such  as  could  be 
funded  by  an  FDTE. 


6.  OPERATIONAL  TEST  (OT). 


a.  General. 


(1)  OT  provides  the  mechanism  v*iere  all  the  key  supportability  elements 
are  brought  together  with  the  materiel  system  under  conditions  of  employment 
most  representative  of  that  expected  when  fielded .  These  user  tests  are 
conducted  during  specific  phase  of  the  materiel  acquisition  process  for 
systems  to  support  scheduled  decision  reviews.  Tests  ere  scheduled  to  occur 
after  sufficient  developmental  testing  (ET)  is  done  to  demonstrate  required 
technical  maturity  of  the  system  and  its  support  elements.  As  such,  Or 
provides  the  "proof  of  the  pudding"  for  operational  effectiveness  and 
suitability  (including  supportability)  of  the  system.  For  the  purposes  of 
this  methodology,  follow-on  evaluation  (FOE)  is  considered  to  be  an 
operational  test,  although  not  so  named. 

(2)  Testing  LOGS  oan  only  be  aooomplished  to  the  degree  that  the  various 
elements  reflect  the  anticipated  application  in  the  field.  In  this  respeot, 
the  maturity  of  test  support  packages  and  the  test  prototype  will  determine 
the  extent  to  which  the  logistics  elements  can  be  implemented/ exercised  and 
valid  data  generated.  The  test  support  paokages  define  the  logistics  ooncept 
for  supporting  materiel.  All  elements  (hardware,  software  and  personnel) 
should  be  exercised  in  a  realistic  environment  as  possible  to  include 
implementation  of  the  support  conoepts  as  defined  by  the  logistics  concept  to 
obtain  logistics  supportability  data. 

(3)  Typically,  there  are  three  phases  during  ^hich  operational  testing 
can  occur  to  validate  the  supportability  of  the  system  in  the  acquisition 


cycle.  Thesa  are  Demonstration  and  Validation  Phase  (OT  I),  Full  Scale 
Engineering  Development  Phase  (OT  II)  and  Production  and  Deployment  Phase 
(FOE).  The  latter  two  phases  (OT  II  and  FOE)  are  when  system  and  support 
maturity  are  such  that  comprehensive  and  effective  LOGS  testing  should  be 
aohieveable.  However,  the  first  phase  (OT  I)  is  vitally  important  to  aiding 
the  materiel,  combat  and  training  developers  in  delivering  this  mature  system 
and  support  in  the  latter  stages.  It  is  here  that  the  system  and  support 
ooncept  is  first  introduced  to  the  environment  (operational  procedures, 
people,  assooiated  equipment,  organization  and  battlefield  conditions)  in 
which  it  must  operate  and  be  supported, 
b.  Testing  Guidelines. 


(1)  OT  I.  The  thrust  of  OT  I  testing  is  to  provide  data  upon  which  a 
mature  system  and  support  elements  oan  be  developed  and  provided  in  the 
following  phase.  Changes  in  the  loglstios  system  and  materiel  design  are  most 
oost  effective  when  identified  early.  As  with  all  testing,  the  OT  I  responds 
to  those  Issues  and  orlterla  assigned  by  the  evaluator  in  determining  the  data 
needs  and  souroes.  Guidelines  for  testing  in  this  phase  are  as  follows: 

(a)  Number  of  Systems  -  Slnoe  they  wil  be  innature  (i.e.,  breadboard, 
brassboard,  advanoed  development  prototype),  quantity  is  not  generally 
orltlcal.  However,  consideration  must  be  given  to  doctrine  and  organizational 
oonoepts.  For  many  systems,  one  or  two  prototypes  will  be  sufficient,  while 


others  may  require  5  to  10  in  order  to  establish  Interface  (e.g., 


(c)  Length  of  Test  -  Based  on  tentative  RAM  requirements  should  provide 


sufficient  iterations  to  validate  mission  profile,  and  failure 
definition/scoring  criteria  and  operation  by  a  sufficient  sample  of  operators 
and  crews  (i.e.,  "as  much  as  possible". 

(d)  Logistic  Concept  -  Preliminary  concept  should  be  implemented. 

(e)  Logistics  Support  Materiel  -  Draft  manuals,  tools  and  test  equipment 
should  be  provided  at  organizational  level  of  maintenance.  As  much  support 
materiel  as  possible  should  be  provided  to  address  DS  support  (especially  for 
critical  DS  maintenance  tasks) .  GS  tasks  should  be  observed  for  their 
complexity  and  equipment  requirements. 

(f)  Logistics  Support  Personnel  -  User  personnel  of  the  planned  MQS  will 
be  used  for  operator  and  organizational  maintenance  (DS  level  if  possible) . 
Training  will  generally  be  contractor  or  developer  provided. 

(g)  Readiness  to  Test  Reviews  -  The  OT  test  agency  reviews  the  SSP 
elements,  operational  test  readiness  statements  (OTRS)  and  safety  release 
prior  to  start  of  test.  When  deficiencies  are  found  such  that  critical  issues 
(including  LOGS  issues.)  cannot  be  addressed,  deoision  makers  are  informed  with 
reccmnended  course  of  action. 

(h)  Obtain  from  Soldier  Support  Center,  the  profile  of  the  qualifications 
for  MOS  test  player  personnel. 

(i)  Specific  skill  requirements  should  be  identified  to  agencies  suppling 
data  collectors.  These  skills  along  with  training  should  provide  an 
individual  who  doubles  as  data  collector  and  on  spot  logistics  evaluator. 

(2)  OT  II.  This  is  normally  the  final  test  prior  to  the  full  production 
decision,  therefore,  comprehensive  testing  of  the  logistics  system  should  be 
accomplished.  Guidelines  are: 

(a)  Number  of  Systems  -  At  least  three  (3).  Doctrine  ana  organization 
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may  demand  more  for  employment  purposes. 

(b)  Employment  -  Tactical  scenario  oriented  employing  system  in 
accordance  with  approved  doctrine,  organization  and  mission  profile. 

Generally  includes  force-on-force  combat  operations. 

(c)  Length  of  Test  -  Based  primarily  on  RAM  and  scheduled  maintenance 
requirements  as  follows: 

1_.  A  minimum  of  three  test  items  will  each  accumulate  test  time  equal  to 
at  least  1.5  times  the  minimum  acceptable  value  (MAV)  for  reliability  and 
operate  past  the  scheduled  organizational,  DS  and  GS  maintenance  points. 

2.  Total  test  time  will  be  sufficient  for  statistical  decision  risk 
levels  specified  in  the  IEP. 

3.  Perform  simulated  maintenance  actions  as  needed  to  accomplish  100%  of 
organizational  tasks,  60-75%  of  DS  and  40-60%  of  GS  when  combined  with  tasks 
required  in  test. 

*(d)  Logistios  Concept  -  Fully  defined  and  implemented  through  GS  level  of 
supply  and  maintenance. 

*(e)  Logistios  Support  Materiel  -  All  logistios  support  hardware  and 
software  should  be  available  and  utilized. 

*(f)  Logistios  Support  Personnel  -  All  operator- maintenance  and  supply  M0S 
personnel  selected  and  trained  in  accordance  with  the  TRADOC  approved  training 
program. 

(g)  Materiel  System  Characteristics  -  Prototype  of  sufficient  maturity 
that  characteristics  that  impact  logistios  represent  design  to  be  fielded. 

(h)  On  system  and  off  system  replace  and  repair  data  will  be  collected. 

(i)  Readiness  to  Treat  Reviews  -  The  OT  test  agency  reviews  the  SSP 
elements,  operational  test  readiness  statements  (0TRS)  and  safety  release 
prior  to  start  of  test.  Wien  deficiencies  are  found  such  that,  critical  issues 


(including  LOGS  issues)  cannot  be  addressed,  decision  makers  are  informed  with 
recommended  course  of  action. 

(j)  Obtain  from  Soldier  Support  Center,  the  profile  of  the  qualifications 
for  MOS  test  player  personnel. 

(k)  Specific  skill  requirements  should  be  identified  to  agencies  suppling 
data  collectors.  These  sills  along  with  training  should  provide  an  individual 
who  doubles  as  data  collector  and  on  spot  logistics  evaluator. 

•These  areas  should  be  the  same  as  that  planned  for  initial  fielding  except 
preouliar  spare  parts  actually  available  may  be  reduced  below  stookage  levels 
as  long  as  developer  is  in  a  position  to  timely  resupply  to  keep  test  on 
schedule.  Example,  if  contractor  support  at  DS  and  GS  levels  is  planned  for 
first  three  years  fielded,  then  the  test  should  include  DS  and  GS  by 
contractor  and  not  the  Army  standard  system.  Likewise,  if  the  standard  Army 
system  is  to  be  used  when  first  fielded,  that  is  the  system  to  be  employed  in 
the  test. 

(3)  OT  I LA,  QT  III  and  FOE.  Theses  tests  answers  those  issues  and 
oriteria  not  addressed  or  unresolved  during  and  verifies  correction  of 
defioienoies  found  in  OT  II.  Therefore,  the  guidelines  are  the  same  as  for  OT 
II  when  LOGS  critical  issues  apply. 

7.  EVALUATION.  Choe  testing  is  completed  and  data  gathered  from  other 
souroes,  the  evaluation  can  be  completed.  Analyses  as  planned  in  the  IEP 
should  be  conducted.  The  evaluation  should  consider  each  issue  for  both 
positive  and  negative  impacts.  Changes  to  the  logistlos  and  materiel  systems 
are  almost  inevitable.  Any  suggested  changes  should  be  thoroughly  examined, 
sinoe  solving  one  problem  often  creates  another.  Conclusions  addressing  the 


480 


r 


--01 


J—  1/1  J/1 


O  <  m  O 

az  s> 

=5  0  O 
O  i— *  * 

1/1 1—  </>  » 

See  o 
=5  Z 
hlil  O  3 

gg>£§ 


o“ 

se^m 
-.3  me 
S^Pc 

o  z 


kl  S  <  UJ 
S.  3  h  ft 
O  1/1  </1  O 


IQ 

* 

O  _J 

o  5 

£. - 22 

ttKXh 

s^gasi 

M  I/)  Q  UJ  UJ  1 
UJLJZ-ICL 

k  j  9  u  o  : 

iii  m  o  >- _ ; 

>  ac  «  o  ac  i 

sw — § 


;  uj  U4  —»cj  uj 

I  J/1  S  O  1/1  UJ  u 

■  fi£  XZOSttZ 

X  O  <  X  «  < 

'  uz  oz 

I  5«!  UJ  *:  O  UJ 

i  o  jFu  P 

:  O  <  Z  O  _l  Z 

:  -j  j—  m  _j  _i  m 

■  O  o<u<< 

i—*  i-2ww2 


>  O  VI 
uj  z  z 

si* 

UJ  >£ 
JHU 

<  X  o 


U>  O  J-I 

m  w  UJ 

tu  B  t/>  m 

m  UJ 

U  is  iK  u. 

UJ  l/J  3  Q  J 
t-  cua.  j  —i 
UO  WOH>-a. 

uj  <  H* 

K  f-ecCwK 

_.ui<25=i0 

owgujc^gS 

KHUJUJmBJZ 

SS0_J*Q.O 
UJ  o 

B.Q  ■  • 

O  <  I—  CVJ 


K—  1/1  m. 
<  O  UJ  w 
2hi« 
H-  t—  I—  X 
U1  t/1  ►— 
mm  5J 

zuyu 

5333 


'  r*  >  v.--  v,'- 


M  TKOMF 


System 

Support 

Package 

Deficiencies 


Logistics 

Support 

Materiel 


Logistics 

Support 

Personnel 


Materiel/ 

System 

Characteristics 


//  1.0 


Logistics 

Support 

Concept 


_L1  Organization 


,  ^  Allocation  of 
-1  »2  Mission 

Responsibilities 


Allocation  of 
Mission  Related 
Hardware/Software 


Allocation  of 

Supply/Maintenance 

Personnel 


1.1.1 

Crew/Operator 

1.1.2 

Organizational 

1.1.3 

Direct  Support 

1.1.4 

General  Support 

1.1.5 

Depot 

1.2.1 

Malnt/Callbratlon 

1.2.2 

Repair  Parts 

1.2.4 

POL 

1.2.4 

Ammunition 

Resupply 

1.2.5 

Joint  Services 
Interface 

1.2.6 

Transportation 

Mission  Related 

1  •  3  •  1 

Ha  rdwa  re/  So  f  twa  re 
Required 

Amount  of  Mission 

1.3.2 

Related  Hardware/ 
Software  Available 

Location  of 

1.3.3 

Mission  Related 

Ha  rdwa  re/Softwa  re 

J :  1  •  1  Crew/Operator 

LliL.  Organizational 
1 « 3  Direct  Support 
i*A»4  General  Support 
L»-4»  5  Depot 


Logistics 

_ n  3.0  Support 

Personnel 


3.1 


Personnel 
Selectl  on 
Requirements 


3.2 


Training 

Program 


3.1.1  Experience 


3.1.2  Training 


3.1.3  Aptitude 


3.2.1  Comprehensiveness 


3.2.2  Level  and  type 
of  Instruction/ 
Program  of  Instruc 


Factors 


APPENDIX  B 


LOGISTICS  BURDEN  ANALYSIS  EXAMPLES 

1.  MANPOWER. 

2.  PETROLEUM,  OIL  AND  LUBRICANTS  <POL). 

3.  TEST  MEASUREMENT  AND  DIAGNOSTIC  EQUIPMENT  (TMDE) . 


MANPOWER  ANALYSIS  EXAMPLE 


Density  -  6000  Items 
Mission  Length  -  12  Hours 
Combat  Usage  Rate  -  2400  Hrs/Year 


Rel 

MTBF 

Failure/Years 

Rqrat 

.97 

500 

4.8 

Data 

.93 

175 

13.7 

Increased  burden  increases  in  Failure/Year  8.9 
Additional  Failures/Years  -  6000  Items  X  8.9  *  53.400  Failures/Year 
2  Manhours/Failure  X  53.400  Failure/Year  s  166,400  Manhours/Year 
EQUIVALENT  to  additional  43  Repairman 


Further  analysis  could  be  performed  to  determine  the  level(s)  of  maintenance 
(i.e.,  MOS)  at  which  the  shortage  ocourred.  Addtionally,  this  shortfall  may 
also  indicate  the  need  for  additional  tools.  TMDE  and  in  the  case  of  contract 

tem,  vehicles. 
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POL  BURDEN  EXAMPLE 


GENERATOR  SET  XYZ 


Data  Required  for  Itoit  Demand  Analysis 


ITEM 

DESCRIPTION 

SOURCE 

POL  CONSUMPTION  PER  SYS 

10  GAL/HR 

OPERATIONAL  TEST 

MISSION  LENGTH 

24  HR/DAY  OPN 

OPERATIONAL 

MODE  SUMMARY 

NO.  CF  GEN  IN  UNIT 

20  GENERATORS 

ORGN  CONCEPT  TSP  ELEMENT 

FUEL  TANKER  CAP 

600  GAL 

TOE  AND  TANKER 

DESCRIPTION 

Unit  Demand  Analysis 

10  GAL/HR  X  24  HRS/DAY  *  240  GAL/DAY/SYSTEM 
240  GAL/DAY/SYSTEM  X  20  SYSTEMS/ UNH  =  4,800  GAL/DAY/UNIT 
4,800  GAL/DA Y/UNJT  :  600  GAL/TANKER  s  8  TANKER  LOADS/DAY/UNIT 

Criteria 

5  GAL/HR/SYSTEM,  2400  GAL/DAY  OR  4  TANKER  LOADS/DAY/UNIT 
ISSUE  CRITERIA  NOT  MET:  ACHIEVEMENT  WAS  TWICE  THE  CRITERIA  THUS 

ANALYSIS  OF  IMPACT  ON  POL  STORAGE  FACILITY 
IS  NEEDED 


POL  BURDEN  EXAMPLE  (CONT). 

Additional  Data  Required  for  Storage  Facility  Impact  Analysis 


ITEM 

TANKER  TURN  AROUND  TIME 
NO,  OF  UNITS  SUPPORTED 

STORAGE  BLADDER  CAP 
TANKER  OPERATIONAL 
AVAILABILITY 
REQUIRED  SUP  ON  HAND 
MANPOWER  STD-BLADDERS 
TANKER 


DESCRIPTION 

SOURCE 

4  HRS 

OPERATIONAL  TEST 

10  UNITS 

LOG  CONCEPT  ELEMENT  OF 

TEST  SUPPORT  PACKAGE 

10,000  GAL 

TOE  A  ITEM  DESCRIPTION 

80S 

FIELD  OPERATIONAL 

READINESS  REPORTS 


20  DAY  STORAGE  FACILITY  STD 

3  OPERATORS/ 10  BLADDERS  ITEM  DESCRIPTION  TOE 

4  OPERATORS/ TRUCK  ITEM  DESCRIPTION 
(X.E.,  2  CREWS  FOR  24  HRS) 


Supply  Faoility  Dnpaot  Analysis 


NUMBER  OF  TANKERS t 

8  TANKER  LOADS  PER  DAY  X  10  UNITS  SUPPORTED  *  80  TANKER  LOADS/DAY  DEMAND 
24  HR/DAY  :  4  HR  TURNAROUND  TIME/ TANKER  «  6  TANKER  LOADS/TANKER/DAY 
80  TANKER  LOADS/DAY  REQUIRED  i  6  TANKER  LOADS/ TANKER/DAY  ■  13  1/3  TANKERS  REQUIRED 
13.3  TANKERS  REQUIRED  X  1.2  OPERATIONAL  AVAILABILITY  FACTOR  ■  16  TANKERS  REQUIRED  IN  UNI 

(3  ARE  DX  SUPPLY  ITEMS  IN  UNIT) 

NUMBER  CF  BLADDERS  REQUIRED: 

4,800  GAL/DAY/UNIT  X  10  UNITS  ■  48,000  GAL/DAY/SUPPLIED 
48,000  GAL/DAY/SUPPLIED  X  20  DAY  SUPPLY  REQUIRED  ■  960,000  OAL  STORAGE  REQUIRED 
960,000  GAL  STORAGE  RE Q  :  10,000  GAL  BLADDER  CAPACITY  b  96  BLADDERS  REQUIRED 


MANPOWER  REQUIRED: 


13  TANKERS  OPERATION  X  4  OPERATOR/TANKER  =  52  TANKER  OPERATORS 
96  STORAGE  BLADDERS  :  10  BLADDERS/3  PEOPLE  X  3  PEOPLE  =  29  BLADDER  OPERATORS 

TOTAL  MANPOWER  *  81  PERSONS  REQUIRED 


Additional  Resources  Required  frao  Not  Having  Met  Criteria 


ACHIEVEMENT  WAS  TWICE  THE  CRITERIA,  THEREFORE,  TWICE  AS  MANY  RESOURCES 
REQUIRED  AS  WOULD  HAVE  BEEN  IF  CRITERIA  HAD  BEEN  MET.  THUS  ADDITIONAL 
RESOURCES  ARE: 


TANKERS  a  8  BLADDERS  «  48  MANPOWER  >  40 

IF  THESE  RESOURCES  ARE  NOT  ADDED,  ONLY  1/2  OF  THE  TOTAL  FLEET  WILL  BE 
MAINTAINED  COMMITABLE  WITH  FUEL  OR  WILL  BE  ABLE  TO  OPERATE  1/2  THE  24  HR  DAY. 
WHILE  OPERATIONAL  AVAILABILITY  (Ao)  DOES  NOT  INCLUDE  MEASUREMENT  OF  FUEL 
AVAILABILITY,  THIS  LACK  OF  RESOURCES  CARRIES  THE  SAME  IMPACT  ON  Ao  AS 
MAINTENANCE  REQUIREMENTS.  IN  EFFECT,  THE  Ao  FOR  THIS  GENERATOR  IS  50*  BEFORE 
MAINTENANCE  BECOMES  INVOLVED  TO  FURTHER  DEGRADE  Ao.  1/2  THE  REQUIRED  FUEL 
RESERVE  WILL  BE  AVAILABLE. 


The  Secondary  Logistics  System 


ALTHOUGH  NOT  ANALYZED,  IF  THE  ADDITIONAL  RESOURCES  ARE  ADDED,  THEN  IT  WILL 
HAVE  THE  IMPACT  OF  DOUBLING  MAINTENANCE  AND  SUPPLY  RESOURCES  REQUIRED  TO 
SUPPORT  TRUCKS  AND  BLADDERS  FOR  POL  SUPPLY  TO  THE  GENERATOR. 
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TMDE  UTILIZATION  EXAMPLE 


Data  Required 


ITEM 

DESCRIPTION 

SOURCE 

NO.  OF  SYS  TO  BE  SPT/UNIT 

200 

TOE 

NO.  OF  SYS  SPT  IN  TEST 

4 

OPERATIONAL  TEST 

DAILY  UTILIZATION  TIME  (TMDE) 

2  HRS/DAY 

OPERATIONAL  TEST 

Ao  FOR  SYSTEM 

.90 

OPERATIONAL  TEST 

Ao  FOR  TMDE 

.80 

DEVELOPMENTAL  & 

OPERATIONAL  TEST 

NO.  OF  TMDE  IN  TEST 

1 

OPERATIONAL  TEST 

NO.  OF  TMDE  PLANNED  FOR  UNIT 

4 

TOE 

Utilization  Analysis 


AVERAGE  UTILIZATION  PER  SYSTEM: 

2  HR/DAY  :  4  SYSTEMS  TESTED  =  1/2-  HR/DAY/SYSTEM 
200  SYSTEMS  IN  UNIT  X  .90  AVAILABILITY  FACTOR  s  180  SYS  OPERATIONAL/DAY/UNIT 
180  SYS  OPERATIONAL  X  1/2  HR/DAY/SYSTEM  =  90  HR/DAY  TMDE  UTILIZATION 

TMDE  REQUIRED: 

24  HR/DAY  X  .80  TMDE  AVAILABILITY  *  19.2  HRS  OPERATION  A VAIL ABLE/DA Y 
90  HR/DAY  TMDE  UTILIZATION  :  19.2  HRS  AVAILABLE/DAY  *4.7  TMDE  ITEMS  REQUIRED 


IMPACT  OF  FIELDING  WITH  4  TMDE: 

1.  WITHOUT  CCMBAT  LOSSES  BEING  CONSIDERED  4  TMDE  CAN  SUPPORT: 

4  TMDE  AUTHORIZED  X  .80  Ao  =  3.2  TMDE  AVAILABLE 

3.2  TMDE  AVAILABLE  X  24  HR/DAY  =  76.8  HR  OPERATION/DAY 
76.8  HRS  OPERATION/DAY  :  1/2  HR  TMDE  DEMAND/DAY/SYSTEM  s  153.6  SYS 
SUPPORTED 

WITH  180  SYSTEMS  OPERATION  PER  DAY,  THIS  SHOWS  A  BACK  LOG  OF  SYSTEMS  WILL 
DEVELOP  AT  THE  TMDE  STATIONS  AND  BUILD  UNIT  ONLY  154  ARE  OPERATIONAL.  THUS 
DEGRADING  OPERATIONAL  AVAILABILITY  OF  THE  SYSTEM. 

2.  CONSIDERING  COMBAT  LOSSES: 

EXPECTED  %  LOST  PER  DAY  *  20% 

THIS  IMPLIES  THAT  AT  THE  END  OF  FIRST  COMBAT  DAY: 

180  SYSTEM  X  .80  =  144  SYSTEMS  +  20  IN  MAINTENANCE  > 

164  SYSTEMS  IN  FLEET  FOR  SECOND  DAY  WITH 

147  BEING  AVAILABLE  (164  X  .90) 

THUS,  AFTER  THE  FIRST  DAY  OF  BATTLE,  THE  TMDE  WILL  BE  ABLE  TO  ACCOMMODATE 
THE  UNIT  SUPPORTED.  CONSIDERING  THIS,  4  TMDE  ITEMS  MAY  BE  ADEQUATE. 
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ABSTRACT.  The  author  would  like  better  to  understand  the  impact  of  v 

missing  data  on  /estimability  (and  variance)  ir^2/ factorials  anc( 2  y  frac¬ 
tional  factflxj^ls.  Hoping  to  find  general izaETlex results,  the  author  ex¬ 
amined  thei [Zy  design  to  determine  what  point*'  could  be  deleted  without 
losing  estiHraroi  1 1  ty  of  main  effects  and  2-faetor  Interactions  (resolution  V 
property).  He  was  guided  by  a  result  . of  P."W.  M.  John  which  shows  that  if  a 

fraction  is  missing  from  a  (l $  design,  then  estimable  effects  are  those 

i-slimablc  from  half  replicates,  and  the  least  squares  estimates  are  obtained  - - - 

by  averaging  the  estimates  from  half  replicates.-^  In  particular,  If  one  or 
two  points  are  missing  from  the  24  factorial,  then  the  remaining  design  is 
of  resolution  V,  and’ the  least  squares  estimates  can  be  written  down  easily 
(without  explicitly  solving  the  normal  equations).  Likewise,  there  are 
essentially  six  ways  a  quarter  replicate  can  be  deleted  from  the  24  design, 
and  only  two  of  those  leave  designs  of  resolution  V.  However,  if  only  three 
points  are  deleted,  the  remaining  design  is  always  of  resolution  V,  esti¬ 
mable  effects  are  not  necessarily  those  estimable  from  half  replicates,  and 
the  least  squares  estimates  of  effects  estimable  from  half  replicates  are 
not  necessarily  averages  of  estimates  from  half  replicates.  The  only  way  to 
delete  four  points  and  fall  to  have  a  remaining  design  of  resolution  V  is  to 
delete  one  of  the  fractions  mentioned  above.  Moreover,  there  are  numerous 
ways  to  delete  five  points  but  still  retain  a  design  of  resolution  V.  The 
author  seeks  insight  to  what  is  going  on  with  missing  points  In  the  24 
factional,  hopefully  insight  which  can  be  generalized  to  other  designs. 

I.  INTRODUCTION.  Factorial  designs  are  frequently  exploited  in  the 
design  of  field  tests  of  military  materiel.  I  suspect  that  they  can  be 
heller  exploited.  For  example,  field  tests  can  be  run  in  blocks  consisting 
of  appropriately  chosen  fractional  factorials  to  reduce  the  bias  due  to  con¬ 
founding  which  is  common  In  much  traditional  field  test  design  (see  Russell, 

1981,  1982).  Unfortunately,  execution  of  a  field  test  seldom  proceeds  as 
planned,  and  rather  large  amounts  of  missing  data  are  common.  I  would  like 
to  be  able  to  produce  experimental  designs  which  are  in  some  sense  robust 
against  data  loss.  In  particular,  I  would  hate  inadvertently  to  use  a 
design  with  nice  theoretical  properties  which  could  easily  be  demolished  by 
missing  data. 

Hoping  to  gain  a  better  understanding  of  the  impact  of  data  loss  in 
factorial  designs,  I  began  an  empirical  study  of  what  I  anticipated  would  be 
a  simple  case,  the  24  design.  (The  24  design  is  also  of  great  practical 
interest,  since  an  experiment  in  four  factors  each  at  two  levels  can  be 
conceived  and  displayed  easily  but  still  provides  substantial  analytical 
richness.)  The  study  was  limited  to  considering  what  points  could  be  de¬ 
leted  from  the  24  design  without  losing  estimability  of  main  effects  and 
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2-factor  interactions  (resolution  V  property)  and  to  considering  the  struc¬ 
ture  of  the  least  squares  (LS)  estimates  obtained.  In  fact,  the  only  non¬ 
trivial  cases  considered  thoroughly  were  the  cases  in  which  three  points 
were  deleted.  Although  I  was  able  to  obtain  some  Insight  in  the  three-point 
cases,  that  insight  was  limited  and  Incomplete.  Since  the  four-  and  five- 
point  cases  appear  to  be  more  complicated,  this  paper  deals  mostly  with  the 
pathology  which  results  when  three  points  are  deleted  from  the  24  design. 

II.  NOTATION.  ANTICIPATED  RESULTS.  AND  ACTUAL  RESULTS.  The  full  24 
design  was  conceived  as  a  labelled  test  point  matrix  (TPM)  in  four  factors 
A,  6,  C,  and  D,  where. the  presence  of  a  particular  lower-case  letter  in  a 
cell  label  Indicated  that  the  corresponding  factor  was  at  high  level  in  that 
cell.  The  labelled  TPM  was 


(1) 

c 

d 

cd 

a 

ac 

ad 

acd 

b 

be 

bd 

bed 

ab 

abc 

abd 

abed 

Potential  data  to  be  obtained  from  this  design  were  modelled  as 
where  the  design  matrix  was 


500 


For  convenience,  the  random  vector  e  was  assumed  to  have  an  Identity  matrix 
as  Its  dispersion  matrix.  Deleting"  points  from  the  TPM  (for  example,  de¬ 
leting  {(1),  a}  and  placing  "X"  at  appropriate  positions  in  the  TPM)  re¬ 
sulted  In  deletion  of  the  corresponding  rows  (for  example,  *C1)  from 

the  design  matrix  to  obtain  a  reduced  design  matrix  R,  and  estimability  of 
all  effects1  was  determined  by  checking  R'jj  for  singularity.  When  R'R  was 
nonsingular,  the  LS  estimate  of  any  specified  effect  was  obtained  from  the 
appropriate  row  of  (R* J8) _1E  and  portrayed  in  terms  of  the  test  points  by 
writing  the  weights  Tor  eacfi  remaining  test  point  in  the  corresponding  TPM: 
for  example,  with  {(1),  a}  missing  from  the  TPM,  the  LS  estimate  of  the  A 
effect  was 


X 

-3 

-3 

-2 

X 

3 

3 

2 

-3 

-2 

-2 

-1 

3 

2 

2 

1 

*  -3b+3ab-3c+3ac-2bc+2abc-3d+3ad-2bd+2abd-2cd+2acd-bcd+abcd. 

I  was  guided  in  this  study  by  a  result  of  P.  W.  M.  John  (1971,  pages 

161-163)  which  shows  that  if  a  fraction  is  missing  a  2n  design,  then  esti¬ 
mable  effects  are  those  estimable  from  half  replicates,  and  the  LS  estimates 
are  obtained  by  averaging  the  estimates  from  half  replicates.  In  particu¬ 
lar,  if  one  or  two  points  are  missing  from  the  24  factorial,  then  the  re¬ 
maining  design  is  of  resolution  V,  and  the  LS  estimates  can  be  written  down 
explicitly  without  solving  the  normal  equations.  For  example,  1(1),  u| 
defines  the  24*2  fraction  with  defining  contrast 

1  =  -B  =  -C  =  BC  =  -0  =  BO  =  CD  =  -BCD 

so  that  If  {(1),  a}  is  deleted  from  the  TPM,  the  remaining  design  contains 
the  half  replicates  I  -  B,  I  =C,  I  =  -BC,  I  =  D ,  I  =  -BD,  I  =  -CD,  and 
I  =  BCD.  The  main  effect  A  is  estimable  in  the  four  half  replicates  defined 
by  two  or  three  factors  (since  it  is  aliased  with  3-  or  4-factor  inter¬ 
actions  in  those  half  replicates),  and  its  LS  estimate  is  obtained  from 


henceforth  "effect"  will  refer  to  the  mean  (I),  a  main  effect  (A,  B, 
C,  or  D)  or  a  2-factor  interaction  (AB,  AC,  AD,  BC,  BD,  or  CD). 


02 


5 


8a 


8a 


8a  8a 


32a 


X 

-3 

-3 

-2 

X 

3 

3 

2 

-3 

-2 

-2 

-1 

3 

2 

2 

1 

I  *  -CD  I  =  BCD 


This  computation2  thus  provides  a  structural  explanation  for  the  rather 
strange  looking  estimate  which  pops  out  of  the  normal  equations. 


There  are  essentially  six  ways  In  which  a  quarter  replicate  can  be 
deleted  from  the  24  design  to  obtain  what  Is  called  the  three-quarter  repli¬ 
cate  defined  by  the  defining  contrast  for  the  deleted  quarter  replicate.' 
Ihese  are  characterized  by  the  numbers  of  1-,  2-,  3-,  and  4-factor  Inter¬ 
actions  In  the  defining  contrasts.  Only  two  yield  designs  of  resolution  V: 
one  Is  represented  by  any  fraction  with  1-,  3-,  and  4- factor  Interactions  In 
Its  defining  contrast,  and  the  other  Is  represented  by  a  design  with  a 
2- factor  Interaction  and  two  3- factor  Interactions  In  Its  defining  contrast. 
The  other  four  designs  fall  to  be  resolution  V  since  not  all  effects  are 
estimable  from  half  replicates  contained  In  the  remaining  design. 

Any  three  points  In  the  24  design  are  contained  In  exactly  one  quarter 
replicate,  and  a  design  obtained  by  deleting  three  points  from  the  24  design 
contains  just  those  half  replicates  which  are  contained  In  the  corresponding 
three-quarter  replicate  obtained  by  deleting  the  quarter  replicate  contain¬ 
ing  the  three  points.  Since  many  of  those  three-quarter  replicates  are  not 


2The  notation  Introduced  In  this  computation  will  be  used  repeatedly  In 
what  follows.  LS  estimates  are  written  as  sums  of  estimates  from  appropri¬ 
ate  fractions.  Each  fraction  Is  represented  as  a  TPM  having  appropriate 
weights  at  all  points  In  the  fraction  and  having  blanks  or  "X"s  at  all 
points  not  In  tne  fraction.  The  defining  contrast  for  the  fraction  Is 
written  below  the  TPM  and  the  expectation  of  the  estimate  Is  written  above 
the  TPM. 


of  resolution  V,  I  therefore  anticipated  finding  many  ways  to  destroy  the 
resolution  V  property  by  deleting  three  points  from  the  TPM.  This  anticipa¬ 
tion  turned  out  to  be  incorrect.  All  designs  obtained  from  the  2 4  design  by 
deleting  three  points  from  the  TPM  are  of  resolution  V:  there  are  estimable 
effects  which  are  not  estimable  from  remaining  half  replicates.  Moreover, 
In  the  two  cases  where  all  effects  are  estimable  from  half  replicates,  the 
least  squares  estimates  are  not  averages  of  estimates  from  half  replicates. 

Examination  of  the  various  ways  four  points  can  be  deleted  from  the  24 
design  showed  that  the  only  way  to  delete  four  points  from  the  24  design  and 
fall  to  have  a  remaining  design  of  resolution  V  was  to  delete  one  of  the 
quarter  replicates  for  which  the  remaining  three-quarter  replicate  Is  not  of 
resolution  V.  Moreover,  there  are  numerous  ways  to  delete  five  points  and 
still  retain  a  design  of  resolution  V. 

III.  DESIGNS  OBTAINED  BY  DELETING  THREE  POINTS  FROM  THE  24  DESIGN. 
There  are  560  ways  to  delete  three  points  from  the  sixteen  points  in  the  TPM 
for  the  24  design.  Since  any  three  points  are  contained  In  exactly  one 
quarter  replicate,  these  560  ways  can  be  classified  Into  six  cases  by  re¬ 
labelling  factors  and  factor  levels  so  that  the  quarter  replicate  containing 
the  deleted  points  also  contains  (1)  and  (1)  is  not  deleted.  These  six 
cases  are  described  In  Table  1. 


TABLE  1. 

Classification  of  designs  Remaining  After  Three  Points  Are  Deleted 
From  the  24  Design. 

Case 

Points  Deleted 

Defining  Contrasts’" 

No.  Ways  Obtained** 

1 

a,  b,  ab 

I  ■  -C  *  -D  ■  CD 

96 

2 

a,  be,  abc 

I  *  -D  *  BC  *  -BCD 

192 

3 

a,  bed,  abed 

I  *  BC  *  BD  =  CD 

64 

4 

ab,  ac,  be 

I  *  -D  =  -ABC  *  ABCD*** 

64 

5 

ab,  cd,  abed 

I  *  AB  *=  CD  =  ABCD 

48 

6 

ab,  acd,  bed 

I  *  -ABC  =  -ABD  *  CD*** 

96 

"defining  contrast  for 'the "quarter  replicate  which  contains  the  points 
deleted. 

**Number  of  ways  this  design  can  be  obtained  by  relabelling  factors  and 
factor  levels  so  that  the  defining  contrast  contains  (1)  and  (1)  Is  not 
deleted. 

***These  two  contrasts  define  three-quarter  replicates  of  resolution  V. 

The  method  used  for  this  reduction  to  six  cases  (relabelling  factors 
and  factor  levels)  changes  signs  and  Interchanges  labels  among  main  effects 
and  among  2-factor  Interactions  but  does  not  Interchange  main  effects  with 


2- factor  interactions.  A  further  reduction  to  three  cases  can  be  accom¬ 
plished  by  introducing  four  new  factor  labels  W,  X,  Y,  and  Z  with  W  =  A,  X  = 
-AB,  Y  =  -AC,  and  Z  =  -AD.  This  induces  a  relabelling  of  the  TPM 


wxyz  wxz  wxy 


c 

d 

cd 

(1) 

ac 

ad 

acd 

4*  A 

wxyz 

be 

bd 

bed 

ID 

x 

abc 

abd 

abed 

wyz 

xz  xyz 


and  produces  a  formal  correspondence  between  the  six  cases  in  pairs  or  two 
(Table  2). 

TABLE  2.  formal  Correspondence  Between  Three-Point  Cases  in  Pairs  of  Two 


Correspondence 


Case  1  ~  Case  3 


ABCD  Notation 


WXYZ  Notation 


\*A 

Case 

1 

3 

Points  Deleted 

a,  b,  ab 

wxyz,  x, 

wyz 

YZ 

Lii 

Defining  Contrast 

I*  -C  =  -D  =  CD 

I  =  WY  = 

WZ  = 

w 

[!i1 

Case  2  ~  Case  5 

i!;.| 

i*M 

& 

Case 

Points  Deleted 

2 

a,  be,  abc 

I  =  -D  =  BC  =  -BCD 

5 

wxyz,  xy, 
I  *  WZ  = 

1  wz 

WXYZ 

Defining  Contrast 

XY  = 

1 

Case  4  ~  Case  6 

i 

Case 

4 

6 

iV 

Points  Deleted 

ab,  ac,  be 

wyz,  wxz, 
I  »  WZ  - 

=  -XYZ 

t 

Del  tiling  Contrast 

I*  -D  *  -ABC  »  ABCD 

-WXY 

In  cases  1  and  3,  one  effect  is  estimable  from  three  half  replicates,  six 
are  estimable  from  one  half  replicate,  and  four  are  not  estimable  from  half 
replicates.  In  cases  2  and  5,  four  effects  are  estimable  from  two  half 
replicates,  three  are  estimable  from  one  half  replicate,  and  four  are  not 
estimable  from  half  replicates.  In  cases  4  and  6,  two  effects  (one  the  mean 
effect)  are  estimable  from  two  half  replicates  and  the  other  nine  effects 
are  estimable  from  one  half  replicate.  Because  of  the  formal  correspondence 
between  the  two  cases  in  each  pair,  structure  of  LS  estimates  need  only  be 
studied  for  the  first  case  In  each  pair  (cases  1,  2,  and  4):  LS  estimates 
for  the  other  case  in  each  pair  (cases  3,  5,  and  6)  can  be  obtained  simply 
by  relabelling  the  weighted  TPMs  for  the  first  case  using  the  WXYZ  notation. 
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I  =  “CO.  AB  is  estimable  from  a1!  three  half  replicates,  A  and  B  are  esti¬ 
mable  only  from  I  =  CD,  AC  and  BO  are  estimable  only  from  I  =  D,  AD  and  BD 
are  estimable  only  from  I  =  C,  and  I,  C,  D,  and  CD  are  not  estimable  from 
half  replicates.  The  LS  estimates  of  effects  estimable  from  half  replicates 
are  the  estimates  from  half  replicates  or  their  average  (in  the  case  of  AB). 
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Although  the  effects  C,  D,  and  CD  are  not  estimable  from  half  replicates, 
each  Is  estimable  from  a  quarter  replicate  which  is  contained  in  the  remain¬ 
ing  design  and  contains  (1);  such  quarter  replicates  will  be  referred  to  as 
(l)-quarter  repl 1cat.es.  The  (l)-quarter  replicates  from  which  C,  D,  and  CD 
are  estimable  are  the  following. 


Effect 

Estimable 


Defining  Contrast 

for  (l)-Quarter  Replicate 


Points  in 


r  Replicate 


I  =  -A  =  -B  =  AB 
I  =  AB  =  AD  =  BO 
I  =  AB  =  AC  =  BC 


(1),  c,  d,  cd 
(1),  c,  abd,  abed 
Cl) ,  d,  abc,  abed 


Inconveniently,  the  estimates  obtained  from  these  (l)-quarter  replicates  are 
not  the  LS  (minimum  variance)  estimates.  Instead,  the  LS  estimate  of  any 
effect  not  estimable  form  a  half  replicate  can  be  obtained  by  estimating  it 
in  a  (l)-quarter  replicate  where  it  is  aliased  with  lAU  (the  effect  esti¬ 
mated  with  smallest  variance  among  those  effects  estimable  form  half  repli¬ 
cates)  then  correcting  for  the  bias  using  the  least  square  estimates  of  the 
effects  from  half  replicates.  For  example,  C  is  aliased  with  -AB  in  four 
(l)-quarter  replicates. 

Points  In  Defining  Contrast  for  Alias  Chain 
(l)-Quarter  Replicate  (l)-Quarter  Replicate  Containina  C 
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aliased  with  effects  estimable  from  more  than  one  half  replicate,  then 
correcting  for  bias.  However,  the  (l)-quarter  replicate  used  must  alias  the 
el  lecl  oi  interest  with  two  effects  estimable  from  two  half  replicates, 
there  are  four  such  (l)-quarFer  replicates  for  each  of  B,  C,  BD,  and  CD,  and 
for  each  of  these  effects,  all  four  (l)-quarter  replicates  yield  the  same 
estimate.  Just  one  of  the  (l)-quarter  replicates  in  each  set  aliases  the 
effect  with  a  higher-order  interaction  (in  this  case,  the  3-factor  inter¬ 
action  ±BCD),  and  the  resulting  LS  estimates  are  as  follows. 


16(B6-of+a 


X  -1 


X  -2 


•2  3 


■2  -1 


-3  0 


2  1 


*1  X 


The  variances  of  these  LS  estimates  are  as  follows. 


Effect  Estimated 


B,  C,  BO,  CD 
I,  0,  BC 
A,  AB,  AC,  AD 


Variance  of  LS  Estimate 


6/32  =  0. 19 
4/32  =  0.12 
3.32  =  0.09 


In  both  case  1  and  case  2  (therefore  also  case  3  and  case  5),  the 
scheme  given  for  obtaining  the  LS  estimates  of  effects  not  estimable  from 
half  replicates  uses  a  (l)-quarter  replicate  and  estimates  from  all  three 
half  replicates  contained  In  the  remaining  design.  However,  using  a  (1)- 
quarter  replicate  and  estimates  from  all  three  half  replicates  contained  In 
the  remaining  design  is  not  enough.  This  and  more  can  be  seen  by  con¬ 
sidering  all  estimates  for  effect  C  in  case  2  which  are  obtained  by  esti¬ 
mating  C  in  a  (l)-quarter  replicate,  then  correcting  for  bias.  For  case  2 
(actually  for  cases  1-6)  there  are  sixteen  (l)-quarter  replicates  in  all. 
In  case  2,  four  of  the  (l)-quarter  replicates  yield  the  LS  estimate  of  C. 
These  are:  I  =  BD  =  -ABC  =  -ACD,  I  =  -A  =  CD  =  -ACD,  I  =  -A  =  -C  =  AC,  and 
I  -  -A  =  -B  =  AB.  The  remaining  twelve  (l)-quarter  replicates  yield  a  total 
of  eight  different  estimates  of  C  having  two  different  variances.  These 
estimates  are  listed  below  together  with  the  (l)-quarter  replicate  user1 
(beneath  the  TPM)  and  the  alias  chain  for  C  in  that  (l)-quarter  replicate 
(nhove  t.hr  TPM). 


C-AD-BD+ABC 


C-AB+D-ABCD 


(each  has  variance  7/32  =  0.22);  thus  merely  Involving  all  three  half  repli¬ 
cates  In  the  estimation  of  C  Is  not  enough.  Each  estimate  In  the  second  row 
estimates  C  In  a  (l)-quarter  replicate  where  Is  Is  aliased  with  both  an 
effect  estimated  from  two  half  replicates  and  a  second  effect  estimated  from 
one  of  those  same  half  replicates;  this  row  contains  two  copies  each  of  two 
different  estimates.  Each  estimate  In  the  third  row  estimates  C  In  a  (1)- 
quarter  replicate  where  It  Is  aliased  only  with  a  single  effect  estimable 
from  two  half  replicates;  each  estimate  in  the  third  row  is  the  same  as  one 
of  the  estimates  In  the  first  row.  All  estimates  in  the  first  three  rows 
have  the  same  variance,  7/32  =  0.22).  Each  of  the  four  different  estimates 
In  the  last  two  rows  has  variance  8/32  =  0.25.  Each  of  the  three  estimates 
in  the  fourth  row  estimates  C  In  a  (l)-quarter  replicate  where  It  is  aliased 
only  with  a  single  effect  estimable  from  one  half  replicate,  and  the  esti¬ 
mate  In  the  last  row  estimates  C  In  the  (l)-quarter  replicate  where  It  Is 
estimable. 

C.  Case  4  and  Case  6  (Represented  by  Case  4).  For  case  4,  the  three 
half  replicates  contained  In  the  remaining  design  are  1*0,  1  -  ABC,  and 
I  =  -ABCD.  0  and  I  are  estimable  from  I  =  ABC  and  1  =  -ABCD.  A,  B,  and  C 
are  estimable  from  I  =  -ABCD;  AC,  BC,  and  AB  are  estimable  from  I  =  0;  and 
AO,  BO,  and  CD  are  estimable  from  I  =  ABC.  Thus  all  effects  are  estimable 
from  half  replicates.  However,  none  of  the  estimates  obtained  by  averaging 
estimates  from  half  replicates  Is  the  LS  estimate.  The  LS  estimate  of  0  can 
be  obtained  by  estimating  D  In  the  (l)-quarter  replicate  where  It  Is  esti¬ 
mable  and  forming  a  stranqely  weighted  average  with  the  estimates  from  the 
two  half  replicates  where  0  Is  estimable. 


On  the  other  hand,  I  Is  not  estimable  from  any  (l)-quarter  replicate;  I  can 
be  estimated  by  estimating  I  In  any  of  the  three  (l)-quarter  replicate:, 
where  It  Is  aliased  with  one  of  the  three  effects  estimable  only  from  1  =  0, 
correcting  for  the  bias,  and  forming  a  strangely  weighted  average  with  the 
estimates  from  the  two  half  replicates  where  I  Is  estimable.  For  example, 
If  the  (l)-quarter  replicate  I  *  AB  =  -ACD  =  -BCD  is  used,  the  LS  estimate 
can  be  obtained  as  follows. 
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A,  6,  C,  AO,  BD,  and  CD  are  each  estimable  from  a  (l)-quarter  replicate 
contained  In  the  design,  and  each  can  be  estimated  by  forming  a  strangely 
weighted  average  of  the  estimate  from  that  (l)-quarter  replicate,  the  esti¬ 
mate  from  a  half  replicate,  and  an  estimate  of  2ero  (the  estimate  of 
-ABC-ABCD  from  I  =■  D).  Those  LS  estimates  fol'low. 
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Finally,  none  of  AB,  AC,  and  AD  is  estimable  from  a  (l)-quarter  replicate, 
but  each  can  be  estimated  by  forming  a  strangely  weighted  average  of  the 
estimate  from  I  =  D  (where  each  is  estimable),  an  estimate  from  I  =  ABC 


(where  each  is  aliased  with  a  main  effect  estimable  from  a  (l)-quarter 
replicate),  and  an  estimate  from  I  =  -ABCD  (where  each  is  aliased  with  a 
2-factor  interaction  estimable  from  a  (l)-quarter  replicate),  then  correct¬ 
ing  for  bias  using  estimates  from  appropriate  (l)-quarter  replicates.  The 
LS  estimates  follow. 
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Of  course,  all  these  LS  estimates  Improved  on  the  LS  estimates  which 
could  be  obtained  If  the  entire  quarter  replicate  I  *  *D=  -ABC  -  ABCD  were 
deleted  to  obtain  a  three-quarter  replicate.  For  D  and  I  the  variance  of 
the  LS  estimate  decreased  from  3/32  *  0.094  to  5/56  *  0.089,  and  for  each  of 
the  other  effects  the  variance  of  the  LS  estimate  decreased  from  1/8  -  0.125 
to  3/28  =  0.107.  Unfortunately,  the  structural  decompositions  of  LS  esti¬ 
mates  obtained  In  case  4  were  not  nearly  so  appealing  as  those  obtained  for 
cases  1  and  2.  They  bore  little  resemblance  to  the  decompositions  In  cases 
1  and  2,  the  ad  hoc  weights  needed  for  the  averages  had  no  obvious  justifi¬ 
cation,  and  the  estimates  of  zero  needed  for  six  of  the  estimates  were 
counter  to  Intuition.  Possibly  there  are  some  other  decompositions  for  case 
4  which  resemble  more  closely  the  decompositions  for  cases  1  and  2,  or 
possibly  there  Is  some  unifying  principle  which  has  been  missed. 

IV.  DESIGNS  OBTAINED  BY  DELETING  FOUR  POINTS  FROM  THE  24  DESIGN.  Four 
points  can  be  deleted  from  the  t*  design' by  deleting  a  quarter  replicate  or 
by  deleting  a  set  of  four  points  which  do  not  form  a  quarter  replicate.  As 
discussed  in  Section  II,  there  are  essentially  six  ways  to  delete  a  quarter 
replicate,  and  four  of  those  leave  a  design  which  Is  not  of  resolution  V. 
The  sledgehammer  approach  was  used  to  examine  the  ways  that  a  set  of  four 
points  which  do  not  form  a  quarter  replicate  can  be  deleted:  appropriate 
points  different  from  (1)  were  deleted  one  at  a  time  from  the  six  cases  In 
Table  1,  and  the  resulting  matrix  jg 1 fi  was  checked  for  singularity.  None  of 
the  exhaustive  list  of  designs  checked  (Table  3)  yielded  a  singular  matrix 
K'R,  which  shows  that  the  only  way  to  delete  four  points  from  the  2*  design 
and  fall  to  have  a  resulting  design  of  resolution  V  is  to  consider  one  of 
the  three-quarter  replicates  which  Is  not  of  resolution  V.  An  attempt  was 
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TABLE  3. --Designs  Derived  by  Deleting  an  Additional  Point  Other  That  (1) 
From  the  Designs  in  Table  1  (All  Are  of  Resolution  V) 


Additional 

Basic  Point 

Case  Points  Deleted  Deleted 


a,  b,  ab 


Additional 

Basic  Point 

Case  Points  Deleted  Deleted 


ab,  ac,  be 


a,  be,  abc 


a,  bed,  abed 


ab,  cd,  abed 


ab,  abc,  bed 


made  to  .describe  for  a  few  of  these  designs  the  structure  of  the  LS  esti¬ 
mates  obtained,  but  the  attempt  was  discontinued  without  success  when  it 
appeared  as  anticipated  that  the  LS  estimates  from  these  designs  were  even 
moro  obstinate  than  the  three-point  cases  considered  in  detail.  Table  4 
gives  the  variances  of  the  LS  estimates  for  each  case  In  Table  3;  these  were 
obtained  as  the  diagonal  entries  of  (B'fi)"1.  Reclassification  of  the  cases 
in  Table  4  by  numbers  of  estimates  having  particular  variances  might  lead  to 
a  considerable  reduction  in  cases  for  detailed  study,  but  I  have  not  yet  had 
time  to  attempt  such  a  reduction. 

V.  DESIGNS  OBTAINED  BY  DELETING  FIVE  POINTS  FROM  THE  24  DESIGN.  Only 
a  few  designs  obtained  by  deleting  five  points  from  the  £4  design  were 
examined.  First,  the  sledgehammer  approach  was  continued  for  designs  de¬ 
rived  from  three-point  case  1.  Appropriate  points  different  from  (1)  were 
deleted  one  at  a  time  from  cases  1-1  through  1-6  In  Table  3  (obviously 
redundant  subcases  were  excluded  from  consideration),  and  the  resulting 
malrix  R'R  was  checked  for  singularity.  Most  of  the  resulting  designs  were 
of  resolution  V,  and  none  of  the  resolution  V  designs  contained  a  quarter 
replicate  within  the  points  deleted.  All  six  designs  which  failed  to  be  of 


resolution  V  contained  a  quarter  replicate  within  the  points  deleted:  for 
four  of  these  designs  the  deleted  quarter  replicate  defined  one  of  the 
three-quarter  replicates  not  of  resolution  V  (that  is,  the  defining  contrast 
for  the  deleted  quarter  replicate  corresponded  to  one  of  cases  1,  2,  3,  or  5 
in  Table  1);  however,  for  two  of  these  designs  the  deleted  quarter  replicate 
defined  a  three-quarter  replicate  of  resolution  V  (corresponding  to  cases  4 
or  6  in  Table  1).  Table  5  lists  the  designs  checked  and  gives  for  all 
designs  which  were  not  of  resolution  V  the  defining  contrast  for  the  quarter 
replicate  contained  within  the  points  deleted  as  well  as  the  corresponding 
case  number  from  Table  1. 

TABLE  5.--Desians  Derived  bv  Deletion  an  Additional  Point  Other  Than  (1 


Additional  Defining  Contrast  for  Corresponding 

Point  Quarter  Replicate  Contained  Case  Number 

Deleted  Resolution  V  Within  Points  Deleted  From  Table  1 


-1 
-2 
-3 
-4 
1-1-5 
-6 


BC  =  BCD 


1-1-7 

acd 

1-1-8 

abc:d 

1-2-1 

ad 

1-2-2 

cd 

1-2-4 

abd 

1-2-5 

acd 

1-3-3  abed 


-BC  =  -BD  =  CD 
CD  =  ABC  =  ADD 


1-5-1  abed 


A  =  CD  =  ACD 


Based  on  the  results  for  designs  derived  from  case  1.  twelve  final 
designs  were  checked.  These  were  the  designs  resulting  when  (1)  was  deleted 
from  cases  4-1  through  4-6  and  6-1  through  6-6  in  Table  3.  These  final 
checks  were  equivalent  to  examining  both  the  resolution  V  three-quarter 
replicate  defined  by  I  =  -0  =  -ABC  =  ABCD  (case  4  in  Table  1)  and  the  reso¬ 
lution  V  three-quart, er  replicate  defined  by  1  =  CD  —  -ABC  =  -ABD  (case  6  in 
Table  1)  to  determine  the  result  of  deleting  each  remaining  point  one  at 
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a  time.  The  TPM  for  each  of  these  three-quarter  replicates  is  given  below 
with  points  in  the  defining  quarter  replicate  labelled  by  "X"  and  each 
remaining  point  labelled  by  (if  deleting  that  point  yields  a  resolution 

V  design)  or  (if  deleting  that  point  fails  to  yield  a  resolution  V 
design). 
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In  each  case,  the  designs  which  fail  to  be  of  resolution  V  are  those  where 
all  five  points  are  contained  in  a  half  replicate  defined  by  a  main  effect 
or  a  2-factor  interaction. 

From  the  cases  studied,  a  promising  conjecture  for  designs  resulting 
when  five  points  are  deleted  from  the  2*  design  in  that  any  such  design  is 
of  resolution  V  If  either  of  the  following  holds: 

I)  The  points  deleted  contain  no  quarter  replicate. 

II)  The  points  deleted  contain  a  quarter  replicate  having  exactly  one 
main  effect  or  exactly  one  2-factor  Interaction  (but  not  both)  In  its 
defining  contrast  and  not  -all  five  points  deleted  are  contained  In  any 
half  replicate  defined  by  a  main  effect  or  a  2-factor  Interaction. 

Unfortunately,  I  have  neither  been  able  to  prove  this  conjecture  algebra¬ 
ically  nor  had  the  time  to  apply  the  sledgehammer  approach  to  designs 
derived  from  cases  2,  3,  and  5  In  Table  1. 

VI.  SUMMARY  AND  QUESTIONS.  In  this  paper,  laborious  computational 
methods  have  been  used  to  investigate  the  designs  resulting  when  less  than 
six  points  are  deleted  from  the  24  design.  Clearly,  deleting  six  or  more 
points  from  the  24  design  leaves  a  remaining  design  which  is  not  of  resolu¬ 
tion  V.  I  had  anticipated  that  there  would  be  many  ways  to  delete  as  few  as 
three  points  from  the  24  design  and  fall  to  have  a  remaining  design  of 
resolution  V.  Instead,  this  paper  showed  that  there  Is  no  way  to  delete 
three  points  from  the  24  design  and  fall  to  have  a  design  of  resolution  V. 
Furthermore,  there  are  many  ways  to  delete  four  or  five  points  from  the  2 4 
design  and  retain  a  remaining  design  of  resolution  V.  (I  conjecture  that 
most  designs  obtained  be  deleting  four  or  five  points  at  random  from  the  24 
cinnlgn  are  nf  ror.olutlon  V,  but  I  have  not  yut  attempted  the  combinatorics 
involved.)  In  the  cases  where  the  least  squares  estimates  were  studied  In 
detail,  however,  the  least  squares  estimates  obstinately  resisted  unified 
structural  representation. 


Like  many  Army  statisticians,  1  am  somewhat  isolated  from  the  statisti¬ 
cal  community:  there  are  few  convenient  persons  with  whom  I  can  discuss 
statistical  problems,  and  1  lack  ready  access  to  an  extensive  statistical 
library.  Fortunately,  this  Conference  provides  an  opportunity  in  the  form 
of  Clinical  Papers  for  statisticians  like  myself  to  present  incompletely 
solved  statistical  problems  for  expert  discussion. 

I  would  like  better  to  understand  the  impact  of  missing  data  on  esti- 

mability  and  variance  In  2n  factorials  and  2n"^  fractional  factorials.  In 
Uh>  context  ol  this  paper  1  hove  live  questions: 

1.  Are  there  blunders  in  this  paper  which  Invalidate  some  or  all  of 
the  results? 


2.  Are  there  known  results  which  encompass  some  or  all  of  the  results 
of  this  paper? 

3.  Are  there  (hopefully  general izable)  methods  which  might  yield  the 
results  of  this  paper  more  easily? 

4.  Have  I  overlooked  some  unifying  principle  which  ties  together  the 
results  of  this  paper? 

5.  Are  the  structural  decompositions  of  least  squares  estimates 
attempted  for  the  three-point  cases  in  this  paper 

I)  the  wrong  ones,  in  the  sense  that  other  more  usable 
versions  exist? 

II)  ii.i I vi' ,  In  Uit<  'jlhisu  Lliul  one  should,  nut  hope  to  ex 
plain  simply  the  solution  of  the  normal  equations 
for  fairly  arbitrary  designs? 


VII.  REFERENCES. 

John,  Peter  W.  M.  (1971).  Statistical  Design  and  Analysis  of  Experi- 
ments.  New  York:  Macmillan. 


Russell,  Carl  T.  (1981).  "The  Potential  Utility  of  Crossing  a  Fraction 
al  Factorial  with  a  Full  Factorial  In  the  Design  of  Field  Tests."  Raleigh: 
US  Army  Research  Office  Repcrt  No.  81-2,  Proceedings  of  the  Twenty-Sixth  Con 
ference  on  Design  of  Experiments.  335-346. 


Russell,  Carl  T.  (1982).  "Selling  a  Complicated  Experimental  Design  to 
the  Field  Test  Operator."  Raleigh:  US  Army  Research  Office  Report  No. 


SENSITIVITY  AND  ASYMPTOTIC  PROPERTIES  OF 
BAYESIAN  RELIABILITY  ESTIMATES 


Barnard  Harris  and  Andrew  P.  Sonus  '  \ 

I 

UNIVERSITY  OF  WISCONSIN  -  MADISON 
MATHEMATICS  RESEARCH  CENTER 


ABSTRACT 

In  decision  theoretic  approaches  to  estimation  problems*  loss  functions 
of  the  type  L(0#0)  ■  10—0 1  ,  o  >  0  are  often  employed of ten  a  «  2.  in 
many  applications  of  reliability  and  life  testing#  such  loss  functions  are 
inappropriate.  Alternative  loss  functions  which  appear  to  be  more  suited  to 
the  intended  application  are  proposed  and  Bayesian  estimates  of  the 
exponential  parameter  are  obtained  for  these. 

Asymptotic  expansions  of  such  estimators  are  given  and  compared  with 
estimators  given  in  the  literoture. 
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SENSITIVITY  AND  ASYMPTOTIC  PROPERTIES  OF 
BAYESIAN  RELIABILITY  ESTIMATES 


Bernard  Harris 
& 

Andrew  P.  Soms 


1 .  INTRODUCTION 


In  decision  theoretic  approaches  to  estimation  problems,  the  error  of 

♦ 

estimation  Is  typically  measured  by  loss  functions  of  the  form 

L(e,§)  ■  |e-6|v,  v  >  o  , 

where  6  Is  the  unknown  parameter  and  §  is  the  corresponding  estimator. 
However,  as  the  following  heuristic  argument  will  demonstrate,  such  a  loss 
function  Is  oten  Inappropriate  for  many  applications  of  reliability  and  life 
testing. 

Let  R,  0  i  R  <  1,  be  the  reliability  of  some  device.  If  Is  an 
estimator  of  R  and  the  true  value  of  R  Is  .5,  then  an  error  of  the  magnitude 
I  ft- R )  ■  .1  would  be  unlikely  to  affect  any  administrative  decision  concerning 
the  feasibility  of  the  device,  since  a  device  whose  reliability  Is  between  .4 
and  .6  would  not  usually  be  regarded  ns  satisfactory.  On  the  other  hfnd,  If 
R  -  .90,  then  one  device  In  ten  falls  and  the  estimate  R  -  .99  would  suggest 
that  only  one  device  in  100  would  fall.  Thus,  one  might  be  Inclined  to  conclude 
that  one-tenth  as  many  replacements  were  needed  as  were  In  fact  required  and 
consequently  seriously  misjudge  maintenance  and  replacement  costs.  A  similar 
but  opposite  error  In  judgement  occurs  when  R  ■  .99  and  R  «  .90.  Consequently, 
it  appears  to  be  desirable  to  concentrate  on  errors  of  estimation  for  R  close 
to  unity. 


Considerations  such  as  those  described  above  suggest  that  loss  function 
suitable  for  reliability  applications  might  have  forms  such  as 

L(R.ft)  =  !R-6!V/(1-R)*c.  0  £  R  <  1 ,  0  £  R  <  1 ,  v  >  0,  k  >  0, 

or 

L(R,R)  «  .  0  s  R  <  1,  0  £  R  <  1 ,  v>0. 

In  order  to  study  the  effects  of  using  such  loss  functions,  we  consider 
a  Bayesian  model  for  exponential  life  testing  and  examine  the  asymptotic  behavior 
of  the  estimates  thus  obtained. 

Specifically  let  X.,,X2,  ...X^  be  Independent  Identically  distributed 
observations  from  an  exponential  distribution  with  probability  density  function 

f(x;8)  ■  9  «fe\  x  >  0,  0  >  0.  (1) 

Then  the  reliability  1"  defined  as 

RT  ■  e'eT  .  (2) 

where  T  >  0  Is  a  constant  assigned  In  advance  of  the  experiment  and  known  as 
the  mission  time.  With  no  loss  of  generality,  we  can  set  T  ■  1,  since  this 
corresponds  to  selecting  the  scale  with  respect  to  which  the  observations 
Xi,X2,...,XN  are  measured. 

In  Section  2,  we  obtain  Bayesian  estimators  for  specific  loss  functions 
of  the  type  described  previously,  when  R  has  a  beta  distribution  prior.  We 
compare  this  with  the  Bayesian  estimator  of  R  that  Is  obtained  by  assigning 
a  gamma  prior  to  0,  a  choice  of  Bayesian  model  quite  often  found  In  the 
literature.  Obviously,  since  R  Is  a  function  of  0,  this  Induces  a  prior 


on  R;  however,  the  priors  assigned  to  6  In  such  studies  do  not  Induce  the 
beta  family  of  priors  or,  R. 

In  Section  3,  the  asymptotic  behavior  of  these  estimators  is  given.  These 
asymptotic  expressions  facilitate  studying  the  sensitivity  of  the  estimators  to 
the  changes  In  the  loss  functions. 

Section  4  Is  devoted  to  comparing  the  results  obtained  herein  with  previous 
estimation  techniques. 

Several  appendices  which  provide  the  technical  details  needed  to  establish 
the  existence  of  the  estimators  and  the  calculation  of  the  asymptotic  expansions 
are  Included  at  the  end  of  the  paper. 


2 .  BAYESIAN  ESTIMATION  FOR  UNCENSORED  AND  TYPE  II  CENSORED  DATA 


Let  Xp  j  s  X^2)  as  •  •  •  sc  X^  be  the  order  statistics  from  a  random 
sample  disbrlbuted  by  (1).  Assume  that  only  the  first  n  order  statistics 
have  been  observed,  1  s  n  s  N.  Then,  it  Is  well-known  that  the  total  time 
on  test  statistic  Y,  defined  by 

Y  ■  ^  X(1)  ♦  (N-n)X(n)  (3) 

Is  a  sufficient  statistic  and  Its  probability  density  function  Is 

fY(y;0)  ■  6n  e"ey  yn"1/r(n),  y  >  0,  9  >  0.  (4) 

To  represent  (4)  In  terms  of  the  reliability  R,  we  reparametrize, 
obtaining 

fy(y;R)  ■  (-log  R)n  Ry  yl1"1/r(n),  0  <  R  <  1,  y  >  0,  (5) 


Let 

T( Rio,P)  •  A*"1  (l-R)®"1,  0  <  R  <  1 .  a  >  0,  6  >  0, 

be  the  prior  distribution  on  R.  Then  the  joint  distribution  of  R  rnd  Y 
Is  given  by 

f(y,Ria,s)  ■  r(a)rie)rjn)  *“+y'1(i-n)9'1(-log  R)n  y"*1, 

0  <  R  <  1 ,  0  <  y,  a  >  0,  8  >  0,  IsnsN. 

We  now  obtain  the  marginal  distribution  of  Y.  Expanding  (l-R)^1  In 
a  binomial  series,  we  obtain 


(6) 


(7) 


Vy;a.s)  ■  rr«-folfon)'  y"'’  f  Ra+y-'H09  «)n  1I0(V>(-1)1r1<‘r*  <8> 

In  (8),  It  Is  understood  that  If  8  Is  a  positive  Integer,  the  series 
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terminates;  that  Is,  all  terms  with  1  >  8  -  1  are  zero. 

The  Interchange  of  summation  and  Integration  In  (8)  Is  justifiable. 
Hence,  we  can  write  (8)  as 

^(yja.e)  ■  r(o)r(S?r(nT  y"’1  J0  (',)i(6i’)  JV^'Vlog  R)n  dR 


T(0+8] 


[WTy  (-D  (  i  )  {0,+y+,)n+ 


r(n+V 


In  particular,  the  Integration  given  in  (9)  Is  valid  over  a  larger  range  of  the 
parameters  than  specified  In  (7).  specifically 


whenever  a  +  y  >  0,  8  +  n  >  0.  These  facts  will  be  subsequently  utilized. 

The  readar  is  referred  to  Appendix  I  for  various  details  relevant  to  these 
remarks  and  calculations. 

Some  particular  cases  of  (10)  are  worth  noting.  If  8*0,  we  have 
fj 

R^y-1  (l-R)"1  (-log  R)n  dR  *  r(n+l)c(n+l,a+y)  (11) 

Jo 

where  c(r,s)  denotes  the  generalized  Rlemann  zeta  function  and  c(r.l)  *  c(r) 
Is  the  Rlemann  zeta  function. 

Combining  the  results  of  (7)  and  (9),  we  can  write  down  the  posterior 
distribution  of  R  given  Y  ■  y  as 


f(R|Y-y) 


(1 


„  R0**"1  (1-R)g~1(-loq  R)n 

r(n+l )  i  (-l)1(e:1)/(a+y+1)n+1 
1=0  1 


This  enables  us  to  calculate  the  Bayes  estimators  for  a  number  of  the  loss 
functions  of  the  type  described  previously. 

For  L|{R,fi)  -  (R-$)2,  the  Bayes  estimator 


i 

\,e(V  "  *<*!**> 


or  equivalently, 


f  Ra+y  (1-R)^"1 (-log  R)n  dR 
_ 

f  Ra+y-1(l-R)B“1(-log  R)ndR 
*#» 


(l: 


(ti) 


I  (-1  )* (^j1  )/{a+y+1+l  )n^ 
1*0 _ \ _ 

l  (-l),(6J1)/(a+yH)IH' 
1>0  ’ 


(14! 


In  particular,  for  g  ■  1 

WLi>  ’ (1  -  s*kr>"+'  • 

Similarly,  for  U  (R,R)  -  (R-ft)*/(l-R)v 

C  |  V 

Is 


C 

v  >  0,  the  Bayes  estimator 


E{R/(l-R)vlY=y} 

E{l/(l-R)v|Y-y) 


I  (-1  >T  *  )  /(a+y+1+1  )nT 

0'8,V  2  I  M)1^1)  /(o+y+1)n+1 

i-o  1 

Finally,  wa  consider  the  loss  function  Lj(R,fi)  ■  ((1-R)"1- 
the  Bayes  estimator  Is  given  by 

i,/,l  ■  i  ■  (eKHl-'Nr'  . 

Since 

f  Ra+y'1(l-R)B’'2  (-log  R)n  dR 
B{  (X-R)“X|v-y)  -  •— 5 - - - 

r(n+l )  I  (-1)W)/(a+yM)n+1 

1-0  1 

and  since  the  Integral  (19)  converges  whenever  0  +  n  -  1  : 

I  <-l)1(8;2)/(»*y+1)n+1 

KtU-Rl^lK-y)  .  -  . 

I  H)1(fl:1)/(a>y+t)n+’ 

1-0  1 

Then,  we  can  write 

l  (-l)*(8J2)/(o+y+1)n+1  -  I  (-l)1!8: 

»  <v.Jd - ! - id - 

0,9  I  (-i),t3^)/(«+yi)n+1 

1-0  1 


0-v+n>O 

(1  -fc)"V.  H< 


0  ,  we  have 


1 )/(o+y+1)n  1 


Applying  the  Pascal  triangle  Identity,  we  have 


I 


Jo  t-D1-1(?:f)/(^)n+1 

Jo'8‘L3’  ‘  V  <-l)V;2)/(o*y+1)"+’  " 

i-0  1 

I  (-1 )1(SI2)/(a+y+1+l )n+1 

„  i-Q _ _ _  *  (22) 

1-0  1 

Comparing  (14)*  (17)  and  (22),  we  have 


^a.B.v^ 


Vb-v(L1* 


and 


^»,B*L3J 


a,B-l 


(Li) 


(23) 


(24) 


Thus,  we  hav«  shown  that  the  three  famlllas  of  astlmators  hava  tha  same 
form.  Hance,  It  Is  possible  to  study  all  of  them  simultaneously  and  a  single 
asymptotic  expansion,  given  in  tho  appendix,  permit*  simultaneous  analysis  of 
their  asymptotic  properties. 

One  caution  should  be  noted.  Me  have  listed  the  posterior  means  whenever 
the  posterior  means  exist.  However,  In  some  of  these  Instances  the  Bayes  risk 
will  be  infinite.  Specifically,  for  the  loss  function  Lg,  the  Bayes  risk 
‘exists  whenever  B  +  n  -  2  >  0. 

In  the  statistical  literature,  one  frequently  finds  the  following  Bayes 
estimator  of  R  (see  for  example,  N.  R.  Mann,  R.  E.  Schafer,  N.  D.  Slngpurwalla 
[6],  p.  398).  This  Is  the  estimator  obtained  by  using  the  loss  function 
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L|(R,ft)  and  assigning  a  gamma  distribution  prior  to  6,  the  parameter  In  the 
exponential  density  (1).  A  brief  sketch  of  the  calculation  of  the  estimator 
follows.  The  prior  Is  given  by 

t(0:y,6)  ■  6y  eY-1  e"69/r(y)  ,  (25] 

where  6,Y  >  0  and  8  >  0.  Then,  calculating  the  posterior  distribution  of 
8,  when  the  data  Is  given  by  the  total  time  on  test  statistic  (3),  It  follows 
that  the  estimator 


*  U4.A  h+ Y 

8<r.«)  ■  ,  (26 

the  conditional  expected  value  of  R  given  Y  ■  y.  Subsequently,  this  will 
be  compared  with  (14),  (17)  and  (22). 

Remark.  To  the  authors,  one  of  the  more  Interesting  properties  of  these 
estimators  Is  the  role  played  by  the  Rlemann  zeta  function  and  the  generalized 
Rlamann  zeta  function.  From  (11),  we  have  that 

8a,o(Ll)  ■  C(n+1,  a+y+1)  /  c(n+l  ,«+y) 


Further,  from  (21),  we  see  that 

Raj(L3)  ■  (c(n+l  ,a+y)  -  (a+y)”n-1 )  /  ;(n+l  ,a+y) 


Rltl(L3)  ■  (c(n+l)  -1)  /c(n+l)  . 


and  If  a  ■  1 ,  y  ■  0, 


Specifically,  If  In  addition  n  ■  1, 

Rltl(L3)  -  (tt2/6  -  1 )  /  (wa/6)  -  1  -  £-  . 

Me  conclude  this  section  with  some  observations  concerning  elementary 
properties  of  the  estimators  given  by  (14),  (17)  and  (22).  With  no  loss  of 
generality,  we  can  examine  specifically  (13)  and  (14).  From  (13),  It  Is 
Immediately  evident  that  0  <  ft  <  1.  Me  now  show  that  6  Is  an  Increasing 
function  of  a  (or  y)  and  a  decreasing  function  of  B,  which  one  would 
naturally  expect  to  be  the  case,  given  the  respective  roles  played  by  a  and 
8  In  the  model.  Therefore,  we  calculate 


f  Ra+y(l-R)M  log  (l-R)(-log  R)n  dR 
J0- 
i 

f  R°‘+y’1(l-R)B'1  (-log  R)n  dR 
Jo 

J  Ra+y(l-R)M  (-log  R)ndR  J  Sa+y-1  (l-Sj^log  (l-S)(-log  S)n  dS 


Thus,  It  suffices  to  consider 


f  Ra+y  (1-R)M  log  (l-R)(-log  R)n  dR  f  Sa*ym) (l-S)0’1 (-log  S)n  d 

J« 

-  f  Ra+y  (1-R)6-1 (-log  R)n  dR  f  Sa+y'1(l-S)B"1log(l-S)(-log  S)n  dS 
Jo 

•  j  j  Ra+y  (l-R)fl’\-log  R)n  Sa+y(l-S)M  (-log  S)n[i2iiil5il3£ll^ 

from  which  It  follows  readily  that  3*1/98  <  o.  Th«  verification  that 
3ft/3o  >  0  (3R/3y  >  0)  Is  completely  parallel  and  Is  omitted. 


^  *>  1 
O  W  O  *  *  *  *  «»  p  V.  *  •  » 


Fi'om  the  results  of  Appendix  III  (see  A. 82),  we  have  that 


(27) 


Ra,B^2,v^ 


ny 


Ra,etL3* 


1  .  1  ,  .1,  (fl-Zle"1^ 

~  ;y 4  nj*<<*z4  * 


>  (1  ‘  ^  ’ 


(28) 


(29) 


As  Is  evident  from  tiie  above*  the  differences  are  small.  For  large  n, 


-1/y 


A  A  VIA  *  f  tf 

Ral8(V/l'<t»,8ltiS,v)  *  *  .ws 

ny*(l-e 


(30) 


and  In  general  gUj)  ~  1  ♦  Otn"1)*  1  s  1*  J  <  3. 

Naturally,  since  the  maximum  likelihood  estimator  Is  e“^y,  one  expects 
the  estimators  above  to  be  asymptotically  equivalent  to  It.  Note  further  that 


va"‘l^/yi(l»e"'IA?) 


Is  bounded  for  all  y  >  0. 

Similarly,  for  the  "traditional"  Bayesian  estimator  (26) ,  the  comparable 
representation  is 

.4. .  4,  +  2<5_+i 

R<y,fi)  ~  e"y  "  "y  .  ****  on 

and  R(*M)/fL  e(L.)  ■  1  +  (Hu1),  i  *  1,2,3.  Thus,  It  appears  that  the 

(*|  P  1 

differences  are  about  as  small  as  could  reasonably  be  expected. 
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4.  SOME  PROPOSED  ESTIMATORS  OF  R 

The  minimum  variance  unbiased  estimator  of  R  has  been  studied  by 
E.  L.  Pugh  [8],  A.  P.  Basu  [2]  and  S.  zaeks  and  M.  Even  [12‘J.  In  our  notation, 
this  Is  given  by 


* 


R 


(  1  -  l/ny)"'1 
0 


ny  >  1 
ny  s  1 


The  asymptotic  representation  for 


R  Is  easily  seen  to  be 


(32) 


1  .  1  _ 1. 


(1  + 


0(n"2)) 


We  can  also  adapt  the  estimators  of  the  exponential  parameter  glve^i  by 
G.  M.  El-Sayyad  [4]  to  correct  them  In  a  naive  manner  to  reliability  estimators. 
Naturally,  the  optimality  hypotheses  used  therein  no  longer  apply, 


R*  ■  exp  ^  ^  ,  p  >  0 

- 1  .  (ML) 

-  e  y  zny  (1  +  0(n-2> ) 


(33) 


(34) 


El-Sayyad  also  provides  an  argument  which  leads  to  the  well-known 
estimator 

§  ■  e^nVny  , 

where  «p(n)  Is  known  as  the  Psl  function  or  digamma  function.  Pron  this,  one 
deduces 

ft  .  ee*(n)/ny  (35) 

and  from  well-known  asymptotic  properties  of  i|»(n),  (see  for  example, 

M.  Abramowltz  and  I.  A.  Stegun  [1]), 


(36) 


1  -  1 


A 

R  - 


J  ^(1  *  0(n"2)  )  . 


the  Psl  function  arises  naturally  in  estimator B  as  the  scale  transformation 
Invariant  estimator  for  squared  error  loss.  In  this  connection  seeT.  S.  Ferguson  [5] 
El-Sayyad  also  obtained  some  Bayesian  point  estimators  for  the  exponential 
parameter  using  the  gamma  prior  and  some  loss  functions  which  are  generalizations 

of  the  squared  error  loss  function. 

In  tho  notation  of  (26).  his  estimators  for  6  are 


@  ' 


-  (6+ny)-1  {r(Y+n+a+B)/r(Y+n+B)}1/0 
■  .'‘,<n+Y>/<«+ny). 


(37) 


(38) 


where  «,$  are  nonnegative  parameters  In  the  loss  functions  used  by  El-Sayyad. 


These  yield  estimators  of  R  as  follows, 


A  - 

R^  *  e  \  1  ■  1 ,2 


and  asymptotically,  we  have, 


(39) 


and 


i  m.  IMi  +  X 

8  .,y  n?  T"F  +  0(n-2)). 


62.;y-^  +  ?iiy  +  sr(1  +  0(J,-2l) . 


(40) 


(41) 


Various  Bayesian  estimators  for  the  exponential  parameter  were  siggested 
;i]  by  S.  K.  Bhattacharyya  [3].  In  one  of  these,  he  considered  the  range  of  t  «  1/0 
to  be  finite  and  used  the  uniform  prior 
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LV 


(B-ct)-  , 


0 


-1 ,  0<OST  $8  , 


otherwise. 


He  also  considered  the  invert#*  gamma  density 


“W/t  (h.j 

e  V 

y  r(v) 


,  0  <  T  <  »,  v.v  >  0. 


Letting 


x 

Y(npc)  ■  |  tn“1  dt 


the  Bayes  estimate  for  (42)  Is 


o  .  K  (nr.1.»Iiy.t31 - i - r  , 

3  Y*(n-1 ,ny)  <1  +  l/ny)n"1 


where  Y*(n,v)  •  Y(n,^)  -  Y  (n,  |)  . 


to  be 


For  the  prior  density  given  by  (43) •  the  Bayesian  estimator  was  snown 


^4  +  ny+y* 


which  asymptotically  behaves  like 


1  u,  1  + 

1L  -  ny  'W  ny  (1  +  0(n-Z))f 


He  also  calculated  the  Bayesian  estimator  for  the  exponential 


prior  obtaining, 


*6- 


.iDiil 

2 


(1  +  -4) 
v  n  y' 


where  the  prior  is  given  by 


S(t)  •  X'1  .*tA  . 


0<x<«,  X  >  0 


It  can  be  shown  that  (47)  may  be  approximated  by 

A  1  “( 0-1 ) 


for  large  n.  Formula  9.7.3  p.  378  In  M.  Abramowltz  and  I.  A.  Stegun  [1] 
may  be  used  to  obtain  more  detailed  asymptotic  Information. 

In  the  papers  by  S.  K.  Slnha  and  I.  Guttman  [10],  [11]  the  Improper  prior 

g(t)  -  t"1 ,  0  <  t  <  *  (50 

Is  assigned  to  t  *■  1/0,  obtaining  as  the  Bayes  •■ttaator 

ft  ■  0  +  l/'ny)“n  ,  (51 

which  yields  the  asymptotic  expression 

-1  .  -1- 

ft  -  e' 9  ’  2nyl  0  +  0(n“2))  .  (52 


V.  M.  Rao  Tummala  and  P.  T.  Sathe  [9]  employed  gamma  priors  and  obtained 
"minimum  expected  loss  estimators"  for  the  reliability.  They  compare  these 
estimators  with  the  Bayes  and  maximum  likelihood  estimators  for  quadratic 
loss  functions  In  estimating  9  . 


'>  •  *  .  "A 


APPENDIX  I 


In  this  appendix,  we  study  the  series  given  In  (9)  and  establish  the  re¬ 
marks  following  (9)  and  (10).  This  can  also  serve  as  a  justification  for  the 
Interchange  of  operations  used  In  deriving  (9).  Specifically,  we  estabMsh 
the  following 


Theorem  A.l.  Let 

k«.8,t)  -  f  (-i)1(8:')(«*i)‘tTt,). 
1**0 

where  a  >  0,  and  t  and  0  are  real ,  and  where 


( 


(0-1) (0-2)  (0-D/1! 

1  . 


1  -  1,2,... 

1  ■  0  . 


(A 


(A 


Then  I(o,0,t)  converges  whenever  0  Is  a  positive  Integer.  If  3  is  not 
a  positive  Integer,  I(oi,@,t)  converges  If  and  only  If  0  +  t  >  0. 


Proof.  Clearly,  If  0  Is  a  positive  Integer,  the  series  (A.l)  terminates 
and  convergence  Is  trivially  verified.  Hence  assume  that  0  is  not  a 
positive  Integer.  Write 


i(a,0,x)  ■  i  (^K-nVir^ 

1<0  ' 

*  i  (e:1H-D1(o+i)-(T+1) . 

i>0  1 

The  first  sum  has  a  finite  (possibly  zero)  number  of  terms.  Hence,  to  study 
convergence,  It  suffices  to  restrict  attention  to  the  second  sum,  which  we 
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write  as 


r  r  Y(Y.l)--.(Y+1-r)(a+i)'*T+1'  , 

lie  «  - n - 

where 


r  ■  1,  Cg  ■  1,  Y  ■  1-8,  If  8  *  0 

r  ■  [8]+1 ,  Cg  ■  (-1  )1*"1  (8-1 ) •  •  *(8-r+l ) ,  Y»(r-8)  If  &>0.(A.5) 


Further,  In  (A.4),  vacuous  products  ar«  intsrprstsd  as  unity  and  y  is  always 
positive. 

We  now  determine  when  (A.4)  converges  and  obtain  estimates  for  the  tall 
of  (A.4),  when  It  Is  convergent.  Hence,  It  suffices  to  consider 

-(t+1  ) 


Ua.Y.r.t)  -  i 

"  1iM  “ 


(A. 6] 


which  has  all  terms  positive.  Note  that  the  sign  of  (A.4)  Is  determl ned  solely 


by  C0. 


Now 


lQg[(VH  M^] 


(Y+1-r)log(Y+1-r)-(Y+1-r)-YlogY  +  Y  +  \  log C(Y) (Y+1-r)] 
-(t+1  )1og(a+1 )  - 1  log  1  +  1  -  j  log  1  -  y  log  (2n) 
■(Y+1-r)[log  1  +  log  (1  +  -p}] +r  -  Y  logY +y  logY  +  ylog  1 

+  y  1og(1  +^j£)-(t+1  )1og  1  -  (t+1  )log(l  +  j)  -  1  log  1 


For  M  sufficiently  large,  we  have  for  all  1  *  M, 


}<(1+y)5e’<e 


Thus, 


where 


1  <  (1  +  j)  <  \ 


1 ,  t  i  "1 

(|)  .  T  <  -1 


Therefore,  we  have  shown  that 


Y-r-i-1 


Id  *  C  .1.  * 


for  some  positive  constant  e  ■  o(Y,r,T).  From  (A. 10),  we  can  Immediately 
observe  that  IM  converges  absolutely  whenever  Y-r-t<0.  Thus,  from  (A. 5), 
we  have  that  If  B  >  0,  I(oi,B»t)  converges  whenever  B+t>0.  Similarly,  If 
B  s  0,  the  same  conclusion  holds. 

To  establish  the  converse,  note  that 


17V 


(TT+J)  (o+1) 


-<T+'Viil 


(Y+1-r)[log  1  +  log(l  +  *^)]  +  r-Y  log  Y  +  j  log  Y  +  £  log  1 
+  j  log  (1  +  ^>)  +  yj[(Y+1-r)"^  -Y’1]  -  (t+1)  log(l  + 

-(t+1)  log  1  -  1  log  1  -  ^  log  1  -  j  log  (211)  -  ^  . 

Now,  for  H  sufficiently  large  and  all  1  >  M 

(Y+1-r)'1  >  0  , 

(121)"1  *  1/12 
and 


(1  +  Ijl)  s  i(Y-r)/(l+(Y-r)) 

Consequently, 

(  V  (v+i)](oi»()-(T+1)/t|  , 
(4.)'V17‘T^ri»-'"-»-\'1,+  T7di(t)  _ 

^  re  dj (t)  is  defined  by  (A. 9). 

Thus 


(A. 11) 


'«  £  k  1Y' 


T-t-l 


(A. 12) 


where  k  -  k(Y.t,r)  is  a  positive  constant.  Thus  lM  diverges  whenever 
Y-r-t  >  0  and  thus  diverges  whenever  b+t  <  0,  establishing  the  theorem. 

B§n»rk.  The  above  analysis  permits  us  to  readily  estimate  the  tall  of  the 
series  whenever  b+t  >  0,  since 


l  <  r 


(H-1 )-{6+X) 

TS+TT 


(A. 13) 
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APPENDIX  II 

In  thi»  appendix,.  we  obtain  a  comp 1 eta  asymptotic  expansion  (n*m)  of 

I  <ct,fl,y)  -  /!  R°+y*  ‘(l-lO^Vlog  K>ndR  , 

n  u 

where  y  *  ny,  y  V  0.  In  particular,  we  establish  the  following  theorem. 
Theorem  A. 2.  For  y  >  0,  o  >  0,  as  n  ♦  for  any  5  >  0,  A  -  2,3,... 

<  IS  !*>»  .*  f/S\ 


.  »'  “  -  /a Vtn( 

..(.-‘‘‘■‘I  . 


where 


N  (8)  «  G(a+ny)  -  n  log  9  -  (8-1 )log(1-e  ) 
n 


and  -8  is  the  largest  positive  root  of  m^(9)  ■  0.  Further, 


n 

V 

rn/(|)l  2r/a,  r  even 

Ur 

^  0  ,  r  odd 


(A. 14) 


(A. IS) 


(A. 16) 


r  even 


Kik)(6)  "  E(“1)  1'Mi3><®) 

nx  n 


1  t 

Htk)(8)/*,i#*tul  31  a»»»kl  k  , 
n  1  K 


(A.  17) 


(A. 18) 


where  the  sum  extends  over  all  *,,»•» **,  >  0  with  t  "  r  • 

3  *  j-3  3 

in  order  to  prove  Theorem  A. 2,  it  is  desirable  to  introduce  a  number  of 
preliminary  lemmas. 

LlPm  A>  1  »  For  j  ■  1#2peeef 

M^'(6)  -  (o+ny)6«.  +  ^  ■1^— li— 3-EIS  ( 6— 1 )  b,.(l-e  ®)  *  ,  (A.1^ 

n  gD  t«o 


whore  6^  -  1,  6^  -  0,  *j  r*  1  and 
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boi  ■  -1'  bii  •  ’>  btj  ‘  °'  *  »  3  or  *  <  0  > 


b»j  -  tb*.3-t  -  “-’>‘*-1.1-1'  o  <  *  «  a,  a  »  a 
Alternatively,  we  can  write 


( A  •  -i  U 


M^j)(6)  -  (a+nyjd^  +  -  {$-1)  \  at;Je“Ae(l-a  "0)~3  , 


(A. 21 


where 


{ 


*n  •  ’•  *01  -  ••  *11  -  0.  1  >  1.  •»,-».*>  1 


*4, 3+1  -  -**»1  -  <3-**'>*t-,,i  .  '  “  <  3.  3  >  » 

Further,  for  j  >  2 


(A. 22 


-  |0-1|e“6<1-e“V3<j-1>l  <  |m(J)(8)|  <  SUrJU.,!, 
e3  n  0j 


♦  |0-1  |o*“®(1-e“0)**3( j-1  }i 


(A.  22 


and  for  n  efficiently  large,  for  each  fixed  0 

a(i-1)»  ,  ,u(j) 


U=iU  <  ,m<3>(9)|  <  . 

ae3  n 


<A.2< 


XSSS&*  (*•!•)#  (A.30),  (A. 21)  and  (A.2S)  are  oaeily  verified,  From  (a. 22), 


it  ie  evident  that  agn  atj  -  <-1)3  ,  J  >  1,  1  <  1  <  j.  Further 


i,  *‘.3*1  -  J,  <-*V  -  j,  <J-it,l*«-1.3 

■  *i  “*‘3  *  ‘’-“V  '  ->  J,  *43  • 


Thus 


i.  *‘3  ■ 


0( j-1  )l 


for  eoae  constant  c.  since  a1t  ■  1,  it  follows  that  c  -  1,  and 


J 


J  •n  "  C-l)3+1(j-l)i  . 

i-0  *3 


(A. 2 


Thus,  from  (A. 25),  we  have 


5 


(A. 26) 


A-1 

(A. 23  and  (A. 24)  follow  directly  from  (A. 26),  establishing  the  lemma. 

Lemma  A. 2.  A  aaquanca  of  asymptotic  (n+«»)  estimates  of  6n  is  provided  by 


6  ■  5  +  0(h  ),  1  ■  I,},... 

n  i 


where 


and 


\  -  V?  , 
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5  ■  B  +  —  (  —  4  —  -  ny  -  ( o+6‘ 

i+1  1  n  U  -5 

Vi  1-.  1 


•') 


(A. 27) 


(A. 28) 


(A. 29) 


Proof*  For  every  c  >  0,  there  is  an  n  sufficiently  large  so  that  *4^(0) 

* 

is  strictly  convex  on  (£,").  Thus,  M^(0)  -  0  has  a  unique  root  &n  on 
(e,“).  (A. 27),  (A. 28)  and  (A. 29)  follow  readily. 

Specifically, 


e*  -  3  +  0(n“’) 


1  +  - (0+0-1)]  +  0 (n  ‘) 

y  n52\1-ri/5  / 

14-J-/ — fill-- -  (0+6-1)]  +  ~r  ( — .  (0+0-1)]  2 

y  ny  \1-#-1/y  /  ny  / 

... .  [— M--  -  (0+0-1)]  +  0(n”3)  . 

n254(l-.-Vn2V  ^  1 


(A. 30) 


Remark i  Applying  lemma  A. 2,  and  letting  6-0  in  (A.23),  we  can  leduce 

n 


n(j-1)l/20*j  <  !M*j>(9*)|  <  2n( j-1 ) \/Q*J  . 

n  n  n  n 


(A. 31) 


Lemma  A. 3.  For  0<r<k,  k>2, 


C<e«'  • 1 


(A. 32) 
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33 


,(r) 

nk 


0,  r  -  1 ,2 


(A. 


lKnK>  <6n) *  <  k(2n)r/3  ?r*  r  “  3,4,. ..,k  ,  {A. 34 

for  n  sufficiently  largo. 

Proof,  from  (A. 18),  (A. 32)  and  (A.33)  follows  trivially.  Therefore,  wo  need 
only  establish  (A. 34).  Prom  (A. 24)  and  (A. 18),  for  n  euffioisntly  large, 

3  <  r  <  k. 


Sinos 


ws  have 


and 


thus 


Now 


Xi 


-  a/v**‘V  3t3,,,)tAk  • 


e  3 


1  -  r 

j*3 


j  ~  ‘  ' 


3  i  < 


j-3 


l  *,  4  r/3  . 


I*nk)<0)l  *  ^'?r  I1/*3,-*-*kl  3  3»««k  n 

8 

lvv*"v  '  X  -  v.j.#v.  «*»s— «i» 

-  T  ir  4 . i,“ 


J-3 

■  r/3 


*3  * 


(A.  35 


m«1 


ml  '3 


4  X  ~7  (log  k)w  »  k  . 


1x^(8)  I  <  k(2n)r/3/8r  . 

From  (A.31),  the  conclusion  follows  rssdily  upon  replacing  6  by  6^  in 
{A. 35). 


Vfs  now  proceed  to  the  proof  of  the  theorem. 

-8 

Proof,  bet  R  ■  e  obtaining 

I  (a,6,y|  -  I’  .-•<«»»?>  . 


(A. 36) 


Choose  tj(n),  T^(n)  so  that 


0  <  x^(n)  <  n/(a+ny)  <  ^(n)  <  *  . 

For  8  >  1, 

«.,«  <  /Vn‘.-».««W9v  <  (’+n?> 


(A. 37) 


(A. 38) 


(A. 39) 


-6 

For  8  <  1,  since  1-e  >  8(i-e  1 )/T^,  0  <  8  <  , 

/o’<n,»«"4e  ‘  (^irf 1  /r  .•e(“+”J».“t#-’d6  . 


whenever 


we  have 


t  <n>  <  *tfcl  , 

o+ny 


t,(n)  /,  -T1lnl\ 


-t  (n)  (a+ny) 
<n)n+P  e  1 


Proceeding  similarly,  wo  have, 


i‘  ...  «(•)«•  <  ((»  r,  .'9l""y,«"d8  , 


where 


f (8)  -  1  ,  8  >  1 


/  -t  (n)\8-1 

■  r  )  * 


8  <  1  . 


From  F.  W.  J.  olver  (7],  p.  70,  we  have 


(A. 40) 
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Hence 


/;  •-9(O^)0n<ae  -  {o+ny )~n**1  /' 


(a+ny)T2<n) 


-u  n. 

«  u  du 


-Ca+ny)T  (n)  +1 

j  (T2(n)) 

( o+ny ) t2 ( n ) -n 


f~x.( »)»9,ie  « 


-<a*n,)rtn) 

f<0)e  (x2(n)) 

<o-fny)T2(n)-n 


(A.  41) 


From  (A. 15)  and  (A. 36),  w«  hava  that 


*Mn(0) 

.(<*#8»y)  -  /ft  a  d6  . 


(A. 42 


Now  writ*,  for  k  >  2  *  .  *  . 

*  <6_0n)  *  (le)  * 

H  (6)  <•  M  (6  )  +  M"(®  )  +•••+  M  « 

n  nn  21  nn  klnn 


whara 


*V"  - 


|,"enlk*'  <*♦,> 

i'W9>i  ‘-fsrrn-'"i  ' ’•«»>' 


(A. 43) 


(A. 44) 


and  C  lies  batwoan  8  and  6  . 
n  n 

From  (A. 30)  or  (A. 15),  it  is  aaaily  saan  that  8  ■  n/(a+ny)  +  o(1)  and 

thus  for  sufficiantly  large  n< 

T  (n)  <  8*  <  T  (n)  . 

l  n  2 

Consequently  for  8  c  (T^n),  T2{n)),  from  (A. 24)  and  (A. 44), 


|Wnk(8)|  < 


(t2 (n)-T^ (n) ) 

u+TTi 


1 2nk  l\ 

\«H 


(A. .45) 


Now  write  g(8)  as 


-m  (8#)  -  e-e* ) 2 


,  «.r  k  <0-0#)A  ,, 


whara  1 


"  ( o  ) ,  Than  writa 
n  n 


k  <6“0*  fit  *  J 

l  -TT- <*’<<') 


#  ^*1^*  '  '*<( 

-  .  (O-e^)  3  *1  <*>•••»!  <®> 

a  r  ( ***  1 )  r  rl  n  n  n  n  rt 

A  rl  g  g  VA  l  w  •  •  i.  I  A*  A. 

r-°  S . V°  3  *  31  3...ki  k 


-1+1  I 


t-i)^3<e-eV  n  (m<3>(8*>\ 
_ \  n  n  / 


4—3  tjf * i 


_ : _ a-3  * 

k  k  A. 

n  41  n  jt  3 

j-3  3  j-3 


whara  tha  aum  ia  ovar  >  0  with  EjA_j  -  A.  Thus,  wa  can  writa 

(A. 47)  aa 


1  *  i  ,tii)<0(9-e*)* +  i  *iJ'<6i>i8V)*  , 


A-k+1 


nk  '  n' 


whara 


-  I  <-uE*3  n  *3/  n  a  i  n  ji*3 

*3».*.**k  j-3  \  "/  j-3  3  j-3 


Accordingly,  wa  dafina 


V91  ■ 


*  Yn  t  3 

"W"  a^0-0*)  ,  *  tt\  a  i 

.  “  ”  2  Ml  *  l  K<A)<8 -eV)  , 


and  consider 


* 

M  ^  0  I 

•  °  "  I^g(8)d8  -  /"„  h  (8)d8|  -  r  . 


**  <  y  “'uT'  s«>-  *  ;0,,(n,»(«id.  *  /tt;(<n"),ihk<..-„.,id9 


+  /v»)®(e>4e * r.M\^i  . 


from  (A*34 ) ,  wo  have 


M  +  I*  *n*  (#"V  1  <  1  +  k  i  <2n)*/3  ?4<e-eV  . 

*■3  Jt—3  n 

Thua  for  t  (n)  >  6*, 

«  n 

M  («*>  .  — < 9-6* J2 

• n  n  v'>*  -  /’  <m  • J  *  (i  ♦  i 

A"3 


Sotting  w  -  ^  (8-e*)2,  wo  obtain 


•vv  t  in>  w>»  •£=/;  >4*4 ) 

“  ^  i  (t.uJ-.V  *-»  ■*  \T»/ 

■  «  n 

Than,  applying  (A.31)  with  j  -  a,  and  (A.41) 


■*fcKV 


*  *  n 


n  Y  \ 


a  '  a'"'  n 


$  a4A/3  nn 

+  k  J3^W  r 


1  ’  ^(T--6*)2  /f Y  .1-1 

<  772=  !•  r  2 


ft  ■ 


k  ,4»/3  [vyo-fr*)  2  ) 

“5",/6  [r  <  vn,-<>2-  -t]  /J 

the  last  Inequality  following  from  inequalities  for  the  incomplete  gamma 
function  on  pages  66  and  70  of  F.  J.  Olver  [7]. 

-  1  +<5  *  *  -  -  -6 

Now  let  n  3  <  T_(n)-9*  -  6*  -  T,(n)  <  n  3  ,  0  <  6  <  ?r  .  Than  for 

2  n  n  l  is 

n  auffioiently  large/  3  <  *  <  k, 


—i 7T  r(T2<n)“VTu,>/V  (r(T2(n)‘en>2  - 


2(4U3)/3k 


<  kn-*/62<44+3)'3 


i|Vn)*vj,t+1>/J(iiVn)-‘;ij 

-  (r  <  V“K>2)  2 


(A. 53) 


Similarly/ 


[iBlv»K>2]’ 2 « [*£  "-m5] ' 9 


-6-2 

■  n  y  /4  . 


Consequently/  there  is  a  constant  o.  -  such  that 

*/y 

|  M  (0  )  |  o.  -  -n3^y3/4 

1  *  "  *•  1  k,y  a  ' 


•vv  /;,<»>  v9>a9l  *  T'jj 

i  a 

n 


Similar  oaloulations  establish  the  same  bound  for 


(A. 54) 
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M  te  >  T  (n) 

•  f0  hk(6)d8  . 


M  ( 9  )  x  (n) 

We  now  oonaider  e  /T  (h^t 8)-g( 9 ) )d9.  From  (A. 46)  and  0 


*  *  2 

M  (9  )  "a^V  ,  IS  ( •  \  o , 

n  n  .-.a.  w  /a%k  —  2  n  ({i  +  l  k'*  (9-9*)  )  . 

i+3  "*  n 


a  n  n  (g(0)  -  1^(9))  -  e 


“wnlt(8) 

(e  -  1)  +  a 


-v9>  j  *<*><,_„*,*  . 

**  nk  n 


A-k+1 


From  (A. 44)  and  (A.31),  for  auffiniently  large  n, 


(r^ni-T^n)) 


k+1 


•wnk(8}|  "“iFrri —  4nk,y 


sinoe  |5  -  -|  <  t  (n)  -  t  (n)  <  2n 

n  *■  2  i 


-1/3 


Thus 


mK  (  6  ) 

|a  nk  -  1|  -  o((T2(n)-t1(n))k+1n) 


and 


,kM. 


|wnkl  -  0((T2(n)  -  T^n))*  n) 
aa  n  +  Further  from  (A.34)« 


I  I  *!j>(0-8n>A|  <  k  l  <2n)V35*(T  (n)  -  T  (n)) 
A-k+1  n  t-k+1  2  1 

k(2n)(k+1)/3  y(k+1)(Ta(n)  -  T1(n))(k+1) 

1  -  (2n)  1//3y(t3(n)  -  T^n)) 


A/3- A, 


Since  n(Ta<n)  -  T^(n))  +0  aa  n  *  ", 


I  l  > <e-e*)A |  <  2k(2n),(k>11/3  5(k+1,(T  (n)  ••  t  ( n ) ) 

A«k+1  ™  n  2  1 


k+1 


.50), 


(A. 55) 


(A. 56) 


(A. 57) 
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for  sufficiently  large  n.  Similarly 


|1  +  I 

i-3  **  n 


<1+1  k(2n>A/'35A(t  (n) 

i-3  * 


t1 (n) ) 


-  1  +  0(t2<n)  -  T1(n))Anl/3) 


Thus 


I1  * 


-  0(<T2<n>  -  T1(n)W‘,B)  . 


Hence 


,Vn) 


-  h ( 6 ) | d6  <  J  2  , 


•  (T2(n)  -  T^n))**1  +  0(VT1)k+1n))de 


0((T2(n)-T1(n))k+1n<k+1,/3}/^2J”>  •  2  <9  #n> 


1 


d6 


o('ta, —  £'*’5 


^  V  •• 


I  (k*1)  <*+D/3-j\ 

l(lj(nl-t1(iv))'kt',a  2] 

.  i|l  .  i  t(kt1)A 


•T  '  '  I 

Prom  (A. 37)  and  (A. 40) 


n+dft  -(ch-nv)T  (n) 

Jn  g(9)d8  <  (T  (n) )  P  a  1 


r(T.(n) ) 


r 


s  <  i 


Where 

and 


0 


Thus 


rtT^n) 


(  1  0  >  1 
1  t  "Vn>^ 

k  ,  J/yn),  0  <  1 


T  (n)  -*  <8  ) 

f0  g<9)d0/s  n  n 


(0-t  (n))(OH*ny)  d 

(yn)) 


tines 


1  2 

*T  <  T  (n)  <  * 
2y  y 

*or  n  sufficiently  large,  wa  can  writs 


de  / 

(Vn,)  pr(t1<nn/M-s  nJ  <  d^y,®) 

whsrs  d1(y,0)  doss  not  depend  on  n.  Similarly, 

f  (0>  -(®fny)Tj(n) 

fT .(„,»(•)■•«  <  -1— V -  , 


(u+ny)Tj{nJ-n 


whsrs 


0,(0)  -  t2(n)f(0)  , 

and  fron  (A*61 ) ,  for  n  sufficiently  large 

V*)  <  d2<?'9>  ' 

doss  not  depend  on  n. 

Mots  further  that 

-  1  -d 

(flHny)x  (n)  -  n  >  (o+n?)(fl%n  3  )  -  a 

•  n  9 


-  0(n 

Therefore,  combining  (A.62),  (A.63),  and  (A.64),  w*  can  writ* 

*  t  (n>  /Tt(n)\  * 

M  (8  )  4*  logCL1  g(8)d8)  <  n  log  - -  -  (8  -T  (n))  (a+ny) 

n  n  0  \8  <n)/  n  1 


+  log  dj(y,8) 


*  m  (x  * 

M  (8  )  +  log (/_  ,„»g(8)48))  <  n  log  ■ -y  -  (8  -T  (n))(a+ny) 

n  n  Vn)  \6  (n)/  n  2 


+  log  d2(y,8)  +  log  n  +  log  c  , 


wh*r*  a  i*  «  suitable  constant.  Accordingly,  w*  consider  ths  expression 
for  i  -  1,2, 


/Vn,\  „* 

n  logl— ; ]  -  (8  -T  (n) )  (a+ny) 

\8  (n>/  n  1 


(  t  (n)”8(n)\  * 

■  n  logll  +  1  ■  t 1  -  (S^t^n) ) (o+ny) 

t . (n)-8  (n)  <t  {n)-8  (n))2\  # 

«  n  — - ; - - - ; -  -  (8-1  (n) )  ( o+ny ) 

8  28  "  1 
5  n  n  1 


*  0< (T1-8*<n) )3 )  . 


How,  since  8  -  —  +  0(n~S,  w*  have 

n  “ 

y 


A  (n>\ 

n  logl  — - - ]  -  (8  -t. 


(n) }  (cH-ny) 


»  n{(Ti(n)-0  (n))(y+0<n  "*) 

(x  <„>-0*(n)>2/  ^ 

-■r-* — (ro<n !) 

-  -  ny( )j 


Thuo 


and 


+  0{(Ti-8*n)3) 


-n 


2y 


.,1 

+  o(n  3  )  • 


(t  (n)-fl  (n)}*  -  4  -4 

*  _ .  » t  _  ■* 


t  An)  -M  J0*)  "  7= 

/g1  g(8)d0/e  n  n  <  •  4y 


1-24 


"M^ ( 8* ) 


/T  ,„,g<0>d0/«  n  n  <  • 


'Vn> 


n 

45 


(A. 66) 


(A.67) 


(A. 68) 


for  n  sufficiently  large. 

Combining  (A. 52),  (A. 54),  (A. 60),  (A.67)  and  (A. 68),  we  have 

M  (8*)  _  iii  _  1  +s 

•  n  ”  l/“  9<0>d0  -  /!*  lyOJdOl  -  0(n  6  3  )  . 


Since 


/T 


-M  (0*) 

-  _n 

(0-0*  )2 

/ 

k 

n  n 

e 

rm  2 

Lm  • 

n 

l1  + 

l 

\ 

i-3 

:> .  . 
_ f  0 

,0-8n)a 

/  l. 

(0 

2  | 

r  K 

l1+  1 
^  U3 

v(l) 

J-m  0 

K  t 

nk 

Y 

* 

-M  (9  ) 
n  n 

V 

/v  r/2 

*  £ 

•4 


i/2 

n 


d9 


r-0 


nk  V  'n 


(A. 69) 


thus  establishing  the  theorem. 
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From  the  results  of  Appendix  11 ,  the  various  estimates  of  sectior.  2  all 


have  asymptotic  estimates  given  by  ratios  of  the  form, 


(A. 70) 


He  now  proceed  to  obtain  an  asymptotic  expansion  for  this  ratio* 
Since  1^(0)  has  the  form 


.*  T. 


e  n  n  2  B  l  VV(6"V 


we  have 


fl 


hk(9)dfl 


“  e 


n  n 


/E  T  r(x)(B*)u  /v  t/2 
Ti/nk  n  *Sr*n 


n  r*0 


where  U  are  the  central  moments  of  the  normal  distribution  with  varianoe 
r 

unity*  Now  N  (6*),  Y  .  and  9*  depend  on  <*.  Thus,  from  (A. 14)  and  (A. 70) 
n  n  n  n 

.  “M  (9*<a+i ,9) )  ♦  m  <9*(a,9))  /  Y  (a,9) 

fi  -  .  n  n  n  n  /  n  • 


"a,®, y 


Y  <a+1,B) 
n 


!  *£**• 


,r/2 


r«0 


^'(9n{a+l,9)yr/(Yn(u+l,fl)  _  +a 


■r 

I  ltiJ)(0*(a,9)wyYn(o,B))r/2 

r-0  **  n  r  n 


-  tel 

(l  +  0(n  6  )) 


(A*  7 1 ) 
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Wc  now  eaploy  the  results  of  Appendix  XI  to  evaluate  (A.71 ) .  From  (A* '5} 

-  M  (8*(a+1,8)  - 
n  n  n  n 


(8*(a,8)  -  8ja+l,6))(a+ny)  -  flJo+1,8) 
n  n  n 


8  (a+1,6) 


♦  n  log  — j" 


8n(a,3) 


+  (8-1)  log 


-6(a+1, 8) 

.  n 

-8*{a,8> 

n 


Using  (A. 30),  direct  calculations  establish 

8*(«,6)  -  8*(a+1,8)  - 

n  n 


J- +  JL.  [HfclL.  ♦  , 

ny2  n*y3  y^-l/y 


-  2(o+8) 


+  Q(n-3)  . 


Similarly, 


8*(«,8)  1  1  /  8-1  1 
T2 - -  1  +  ~  +  -l-(  — er,L.:  .  (0+6-1) 

8n(«+l,8)  ny  #V\1.,*Vy  j 


Thus, 


n  log(8*(«,8)/9*(a+1,8>)  - 
ii  n 


japL.w 

y  »y  \,.,-'/y 


/  ny3(l-e'1^). 


(A. 721 


(A»73 


♦  0(n"2) 


Mow  write 


-e„«»,®) 

i-a  n  -  (i 


.  *  Clfl  Li fcu. . 


(a+0-1  )|  +  0(n’2) 


and  raplacing  o  by  o+l  in  (A.76)  producas  1**a 
Rawrita  (A. 76)  aa 


-0  (o+1,0) 
n 


♦ - — — (—5^ - +  0(n"J>]  . 


Than  ,  # 

-6  (a,0) 


log  1-.  ^  .10,(1-.-^,  *_t 


-i/y 


ill 


Thus, 


log 


conaaquantly. 


-6*(a+l,0)\ 

l-a  n  I 


-1/y>;2  \i-.’1/y 


-  (0+0-1  )] 


n<  1-a  7  )y' 


0(n~2)  . 


-a‘1/y  -2 

— - ,+  0(n  d)  . 


-2  ,  -i/y) 

y  1-o  *) 


ny 


M  (O*(o,0)  -  m  (6*(o+1,0))  - 
n  n  n  n 


.l.iL.li.Jtlll 

y  ny 


-1/y 


+  0(n“2)  . 


n52(l-.“Vy) 


Ha  now  avaluata  Y  (0,0) A  t 0+1,0).  sinea 
n  n 


i  b-i  17  “#*ca»#,\"a  /  •e«(a»Mv 

-v-”  -  tot -^b  )  •  (’-  ) 


from  (A. 30) 


(9*(a,0))"2  -  y2  1  + 
n 


2_  /  B-1  _ 

ny  \  «__-Vy 


(a+0-1  m"1  +  0(n“2) 


8 


0(n”2)  , 


wa  sat 


Y  <a,0)/Y  (0*1,0)  -  1  -  —  +  0(n"2) 
n  n  ny 


(y  (o,0)/Y  (a*1 ,0))2  -  1  -  0(n“2) 


finally,  wa  aat  k  -  5,  obtaining 


I  K^>(e*(a,fl))wr/(Y^«,6))r/2 


-  1  -  Hi4,(8ja<P))/8Y2(o,e)  . 


n  n 


frost  U.  19),  (A. 30)  and  (A. 79),  wa  sat  that 


1-Mi4>(O*(a+l,0))/8Y2<a+1,3) 

n  n _ _ _ n _ 


1“H^4)(6*(a,0))/8Y2(a,ff) 


-  1  +  0(n”2) 


n  '  n 


Containing  (A. 71),  (A. 76),  (A. 77),  (A. 78),  (A. 79)  and  (A. 80),  wa  hava 


1  +  JL  (B-1  )~Vy 

-  -2{0  2  “  -2(  -1/y, 

y  ny  ny  vl-a  ' ) 


o,8,y 


•~)(l  +  0(n“2))  . 

ny 
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A  NOTE  ON  THE  VARIABLE  KERNEL  ESTIMATOR 
OF  THE  HAZARD  FUNCTION  FROM  RANDOMLY  CENSORED  DATA* 
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ABSTRACT 


V  In  a  recant  paper  (Tanner  and  Wong  (1982b)),  a  family  of  data-based 
nonparametric  hasard  estimators  was  introducsd.  Sevaral  of  these  estimators 
were  studied  in  an  extensive  simulation  cxperimsnt.  The  estimator  which 
allows  for  variable  bandwidth  was  found  to  have  a  superior  performance. 

In  this  note,  sufficient  conditions  for  the  variable  kernel  estimator  to 


be  strongly  consistent  are  presented. 
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1.  INTRODUCTION 


Let  (T^C^),  i  •  l,...,n  be  independent  and  identically  distributed 
pairs  of  non-negative  random  variables.  Assume  that  and  are  in¬ 

dependent  for  all  i.  penote  by  ST(fT)  and  Sc(fc)  the  survivor  (den¬ 
sity)  functions  of  Tj  and  respectively.  (See  Hiller  (1981).)  In 

the  random  censorship  model  we  observe  the  pairs  (Y1,?i),  1  -  l,...,n 

where 


-  min(T^,C1) 
<  C£) 


The  problem  is  to  estimate  the  hazard  function  h(z) 
Define  as  the  distance  from  the  point  z 
of  Y .  , . . .  ,Y .  ,  where  <S .  ■  2>,  ■  . . .  ■  6.  -1 

*1  H  11  l2  H 


•  fT(z)/ST(z). 
to  the  kth  nearest 
(assume  k  <  i) .  R^, 


then,  is  the  distance  to  the  kth  closest  failure  neighbor  from  z.  De¬ 
fine  J  to  be  the  index  of  the  largest  order  statistic  of  the  Z  failure 

,  s  +  RkJ.  (If  z- 

let  J  •  i^-1.)  Let  J*  »  min(n,J  +  k)  and  let  6^  be  the  indicator 
random  variable  associated  with  Y^. 

The  variable  kernel  estimator  of  h(z)  is  defined  as 


<  Y 


points  which  prscadss  the  Interval  [z  -  R^ 


h  (z) 
n 


t>66 


n-i+1 


(1) 


This  estimator  has  the  appealing  feature  that  the  configuration  of  the  data 
plays  a  role  In  determining  the  degree  of  smoothing.  In  data  sparse  (dense) 
regions,  will  be  large  (small)  and  the  kernel  will  be  flat  (peaked). 

In  an  extensive  simulation  study,  Tanner  and  Wong  (1982b)  compare  a 
data-based  3-parameter  nonparametrlc  estimator,  which  incorporates  the  kth 
nearest  failure  neighbor  distance,  to  a  data-based  1-parameter  nonparametrlc 
estimator. with  constant  bandwidth.  (The  theoretical  properties  of  the  1- 
parameter  estimator  are  discussed  in  detail  in  Tanner  and  Wong  (1982a),  while 
Yandell  (1983)  and  Ramlau-Hansen  (1983)  examine  a  truncated  1-parameter 
kernel  estimator.)  The  performance  of  the  data-based  3-parameter  estimator 
is  shown  to  be  superior  to  that  of  the  1-parameter  estimator.  Our  ultimate 
goal  Is  to  establish  the  theoretical  properties  of  this  fully  data-adaptive 
estimator.  However,  this  is  a  difficult  problem.  We  regard  the  present 
paper  as  solving  a  significant  component  problem.  One  must  understand  how 
these  estimators  behave  whan  the  parameters  are  chosen  deterministically 
as  a  prerequisite  to  the  analysis  of  the  behavior  of  the  data-adaptive  procedure. 

Several  authors  (Fix  and  Hodges  (1951),  Loftsgaarden  and  Quesenberry  (1965), 
Wagner  (1975),  Moore  and  Yackel  (1977)  and  Mack  and  Rosenblatt  (1979))  have 
discussed  the  theoretical  properties  of  the  variable  kernel  estimator  of  the 
density  function  and  the  special  case  nearest  neighbor  estimator.  We  point  out 
that  the  estimation  of  the  hazard  is  a  somewhat  more  difficult  problem,  since 
formula  (1)  depends  on  both  the  order  statistics  of  the  sample  and  the  ordering 
induced  by  estimating  the  hazard  at  a  point  and  sorting  the  data  to  obtain  the 
kth  nearest  failure  neighbor  of  this  point.  For  this  reason,  direct  application 
of  previous  techniques  yields  intractable  formulas. 


2.  CONSISTENCY  OF  h 
■ - •  n 

We  assume  that  the  survivor  and  density  functions  are  continuous  in 
a  neighborhood  around  the  point  of  interest.  We  begin  with  some  lemmas. 
In  Lemma  2.1,  we  present  the  density  of  R^.  We  use  this  result  in  Lemma 
2.2  to  show  that  R^  converges  almost  surely  to  zero.  Lemma  2.3  enables 
us  to  use  Proposition  31  of  Asian  (1978)  to  prove  almost  sure  convergence 
of  hn(e). 

LEMMA  2.1.  Let  R^  represent  the  distance  between  the  point  x 
and  its  kth  nearest  failure  point.  Let  p  -  F(T^  >  C^), 

G(r)  "  ;|x-y|<r  *T(y)sc<y>dy.  *<*>  -  (l-P)C<r), 

G’(r)  »  fT(x-r)Sc(x- r)  +  fT<x  +  r)Sc(x  +  r)  and  F'(r)  -  (l-p)G’(r). 
Then  tha  density  of  R^  Is 

fj^  -  n  ( k-1 )  p(r)k“1(l-F(r))n"kF,(r)  . 

PROOF.  The  probability  of  m  censored  observations  in  a  sample  of 
size  n  is  given  as 


p<»>  ■  (”)p-u-p)n-m  . 

In  addition,  given  that  m  observations  in  a  sample  of  size  n  have  been 
censored,  the  density  of  is  given  so 
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J 


l, 


P(r|»)  -  <n-m)  G(r)k"1(l-G(r))n",n“k  G'(r)  . 

The  result  now  follows  by  direct  calculation. 

LEMMA  2.2.  Let  k  »  k(n)  -  [n°],  0  <  0  <  1,  and  let  R^  be  defined 

as  above.  Then  R^  0. 

PROOF.  Given  6'  >  0.  by  Lemma  2.1  and  repeated  application  of 
Integration  by  parts  It  is  easy  to  show  that 

PO^  >  i’)  <  kJ1  jfi1(l-6)n'i 

From  Chernoff  (1952) ,  it  can  be  shown  that  this  quantity  is  bounded  by  2-nA^ , 
where  A(r)  equals 

-  [fc^d'u-M1*4]  +  lo,2  [p'd-p)1-"]  ♦  1".2(^)P  -  1.8 i^y, 

with  p  -  k/n.  It  Is  now  straightforward  to  show  that 

-nA(n)  -  -  |log2  ^1  -  -ij)  |  J-  (l-a>  log2(nc)  -  |log2^l  - 

For  r.  sufficiently  large  we  have  -nA(n)  <  -no',  for  some  positive 
e'  <  e,  and  the  result  follows. 

LEMMA  2.3.  Let  R^  and  k  *  k(n)  be  defined  as  above,  with 
1/2  <  o  <  1.  Then 


1/2 

n  _  a.s. 

log(n)  R<n) 


.  .  *.•  »  .  k 


PROOF.  The  result  will  follow  If  we  can  show  that  for  all  t  >  0, 

~  /  1/2  \ 

J2  P  “kOO  5  E f  *  +  "  * 

Mow 

F(en) 

p(lSiTST  *k(»)  s  £)  '  /  "  (tl)  t>n'k'“. 

where  e  ■  e  $)•  .  One  can  show  that  the  result  will  follow  If 

»  nl/2 

„  /n 

l  f  tk_1(l-  t)n"kdt  <  +». 

nm2  o  '  ^ 

Proceeding  analogously  to  Lamia  2.2 

/“»(&)  .‘"W**.  -  Jk(l)EJa'En)n"1  ‘  2_nA<n) 
0 

where,  for  p  ••  lt/n, 

*<n)  -  -log2  ep  -  log2(l-  en)1"*P  +  log2(p)p  +  log2(l  -  p)1”15 


Hence  for  1/2  <  a  <  1  and  sufficiently  large  n,  -nA(n)  <  -n  ,  where 
0  <  a’  <  a,  and  the  result  follows. 
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THEOREM  2.1.  Let  k  -  k(n)  -  [n  ],  1/2  <  o  <  1,  and  be 

defined  es  above,  let  K(<)  be  a  function  of  bounded  variation  with  com¬ 


pact  eupport  on  the  Interval  J-1,+1],  let  h  be  contlnuoue  at  a,  then 


h  (a)  h(a). 

tk 
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.  Let  An  -  {suplh.Cs)  -  h(a) |  >  e>  for  e  >  0.  How  chooee 


min 


6  such  that  (s-2S)i  0,  then 
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and  by  Lemma  2.2,  \  — v  0.  Hence  we  need  only  consider  the  event 

{A  fl  {sup  R^^  <  S}}.  How  by  the  triangle  Inequality  one  can  show  that 
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and 


Regarding  the  first  event ,  using  an  application  of  Integration  by 
parts,  one  can  show 


<  5 


))  c  b( 
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since  K(*>  is  assumed  to  be  a  function  of  bounded  variation  with  compact 
support.  But  by  Proposition  31  of  Asian  (1978)  and  Lenna  2.3,  we  have  that 
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Regarding  the  second  event,  it  is  immediate  that 


i'n{ 


-  ■  ■  1 2  ” 4 


}}  =  • 


Therefore,  If  the  function 


f(a)  - 


14. 


K(u)h(s  -  2au)du  *  h(s) 


a  ■  0 


a  >  0 


is  continuous  at  s,  then  lie.  P(A' ')  »  0,  since  R.  * ’*'■*  o,  Row  f(a) 

n  * 

can  be  shown  to  be  dominated  by 


max  |h(s  -  2au)  -  h(s) |  f  |K(u)|du. 
|u|Sl  J \.A* 


u|<l 


If  we  let  a  0,  the  result  follows. 
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THE  EARLY  INFLUENCE  OF  W.  EDWARDS  DEMING 
ON  THE  DEVELOPMENT  OF  STATISTICAL  QUALITY  CONTROL 
IN  THE  UNITED  STATES  AND  IN  JAPAN 


Nancy  R.  Mann 
Department  of  Biomathematics 
UCLA 

Los  Angeles,  California  90024 


The  first  time  that  I  heard  a  detailed  account  of  W.  Edwards  Deming's 
early  experiences  with  quality  control,  I  was  in  Washington  to  attend  an 
"International  Conference  on  Fatigue  Failure  of  Engineering  Structures"  and 
to  give  a  paper  there.  It  was  1969  and  the  69th  year  of  his  life. 


Some  time  before  the  conference,  I  received  ar  invitation  to  join  Dr. 

Deming  and  another  conference  speaker  who  was  from  the  University  of 

Lisbon.  The  invitation  waB  for  dinner  at  the  Cosmos  Club  on  the  first 

evening  of  the  event.  I  accepted  with  some  feeling  of  anticipation,  and 

then  when  I  arrived  In  town,  touched  base  by  phone  to  find  when  I  should 
I 

meet  them  for  the  occasion. 

At  that  time  I  discovered  two  problems  concerning  my  projected  fellow 
dinner  guest.  First,  he  was  not  the  Portuguese  professor  whom  Ed  Deming 
had  met  some  years  before,  but  a  younger  man  with  the  same  unusual  last 
name.  Second,  he  had  stopped  in  New  York,  on  his  way  to  Washington 
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from  Portugal,  to  consult  with  a  co-investigator  at  Columbia  University;  had 
thereupon  contracted  food  poisoning,  or  perhaps  stomach  flu;  and  had 
rushed  back  home  to  Lisbon  to  ieek  treatment  or  comfort  or  both.  He  didn't 
make  it  to  the  conference,  though  he  did  later  submit  a  paper  for  the 
published  proceedings. 

So,  it  didn't  matter  that  this  was  the  wrong  man;  he  didn't  come  to 
dinner  anyhow.  That  left  the  two  of  us  to  eat  and  converse,  once  we  met 
in  the  Ladies'  Parlor,  just  inside  the  ladies'  entrance  to  the  Cosmos  Club. 
(In  those  days,  my  consciousness  of  male  chauvinism  was  languishing 
comfortably,  yet  to  be  raised,  so  I  paid  little  attention  to  this  quaint 
arrangement.)  I  might  add  that  the  ladies'  entrance  is  still  there  at  the 
Cosmos  Club,  but  its  use  has  been  abandoned  for  reasons  of  security. 

Mealtime  provided  a  chance  for  me  to  find  out  how  W. Edwards  Deming, 
who  was  originally  trained  In  mathematics  and  physics,  had  made  such  an 
impact  on  the  discipline  of  statistical  quality  control  and  had  had  so  much 
infuence  in  its  application  in  tnis  country  and  in  Japan. 

Recently,  I  have  refreshed  my  memory  and  filled  in  details  in 
conversations  with  him  during  several  Saturday  and  Sunday  afternoons  in 
his  home-based  office  in  Washington.  These  took  place  between  his  trips  to 
South  Africa,  British  Columbia,  the  Netherlands,  Japan,  Korea  and  most  of 
the  major  and  many  of  the  minor  cities  of  the  United  States.  I  have  also 
been  aided  in  the  following  by  documentation  provided  (directly  or 
indirectly)  by  Churchill  Eisenhart,  Allen  Wallis,  Holbrook  Working  and  THE 
MAINICHI  DAILY  NEWS  of  Tokyo,  the  issue  of  November  10,  1965. 

It's  useful  to  begin  the  story  in  March  of  1938,  shortly  before  the  time 
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Ed  Deming  left  his  position  as  a  mathematical  physicist  at  the  U.  S. 
Department  of  Agriculture  (USDA)  to  join  the  U.S.  Census  Bureau.  At  that 
time,  he  arranged  for  Dr.  Walter  Shewhart  of  the  Bell  Telephone 
Laboratories  to  deliver  a  series  of  four  lectures  on  "Statistical  Method  from 
the  Viewpoint  of  Quality  Control"  at  the  USDA  Graduate  School.  These 
lectures  were  published  by  the  Graduate  School  in  1939  "with  the  editorial 
assistance  of  W.  Edwards  Deming". 

Shewhart,  in  his  1931  book,  "Economic  Control  of  Quality  of 
Manufactured  Product,"  had  given  his  criteria  for  determining  whether  a 
given  set  of  numerical  data  was  in  a  state  of  statistical  control  --  and  had 
given  also  the  particulars  of  his  corresponding  control-chart  techniques. 

In  a  1981  interview  published  in  MILITARY  SCIENCE  AND  TECHNOLOGY, 
Volume  1,  Issue  No.  3,  Ed  Deming  discussed  Shewhart's  important 
contribution. 

"Dr.  Shewhart  first,  saw  the  fact  that  random  variation  represents 
the  ultimate  state  of  a  system,  that  when  you  have  achieved  that 
state,  then  you  have  an  identifiable  process,  and  until  then  you  do 
not  --  you  have  chaos  in  a  small  degree  or  to  a  high  degree. 

"Causes  of  nonrandom  variation  are  called  assignable  causes  or 
special  causes.  And  those  are  usually  chargeable  to  particular,  local 
conditions  that  the  workers  can  recognize  and  eliminate.  And  then 
you  have  left  random  variation  that  defines  the  system,  and  from  then 
on  only  the  management  can  improve  it.  That  was  Shewhart's  great 
contribution." 

In  his  first  book,  Shewhart  was  concerned  with  the  application  of  his 


methods  and  techniques  in  controlling  the  quality  of  industrial  production 
processes.  In  the  USDA  lectures  and  the  book  derived  from  them, 
however,  he  not  only  reviewed  his  earlier  work  and  the  developments 
during  the  intervening  years,  but  also  devoted  one  full  lecture  (chapter)  to 
their  application  to  the  results  of  measurement  of  physical  properties  and 
constants,  and  one  lecture  (chapter)  to  the  "specification  of  accuracy  and 
precision"  of  measurement  processes  generally. 

The  editing  of  the  Shewhart  book,  along  with  earlier  work  with  Harold 
Dodge  at  Bell  Labs  and  Captain  Leslie  Simon  (later  Lt.  General)  at 
Aberdeen  Proving  Ground,  had  a  profound  effect  on  W.  Edwards  Doming. 
The  ideas  that  resulted  from  this  exposure  are  central  to  his  total 
philosophy  of  dealing  with  problems  of  production. 

He  first  made  use  of  the  material  in  the  two  chapters  in  the  Shewhart 
book  on  measurement  and  precision  in  consulting  he  did  some  few  years 
later  for  tha  U.S  War  Department  during  World  War  II.  Shewhart’s  general 
theory,  however,  he  applied  shortly  after  he  became  familiar  with  it.  This 
is  explained  by  Dr.  Churchill  Eisenhart,  Senior  Research  Fellow  at  the 
National  Bureau  of  Standards,  in  notes  he  wrote  recently  on  Deming 
accomplishments.  The  notes  were  to  be  read  on  the  occasion  of  the 
presentation  of  a  fourth  honorary  doctorate  to  Dr.  Deming,  this  by  the 
University  of  Maryland  on  January  8,  1983.  Many  of  the  facts  were 
obtained  from  "Revolution  ir.  U.S.  Government  Statistics,  1926  -  1976,"  a 
1978  U.S.  Government  Printing  Office  publication  by  Joseph  Duncan  and 
William  Shelton. 

"In  neither  of  his  books,  nor  in  his  other  related  publications,  did 
Shewhart  mention  or  hint  that  his  statistical  quality  control  procedures 


could  be  applied  equally  well  to  routine  clerical  operations,  with 
comparable  beneficial  effect.  This  if  obvious  once  one  thinks  of  it, 
and  think  of  it  Deming  did.  Statistical  quality  control  procedures  were 
applied,  at  his  suggestion,  in  the  clerical  operations  of  the  1940 
population  census,  for  example  in  the  coding  and  card-punching 
operations.  The  procedure  worked  very  well.  During  the  learning 
period,  the  error  rate  of  a  card  puncher  was  high;  but  with  training 
and  experience,  a  good  card  puncher's  error  rate  dropped  markedly 
and  exhibited  statistical  control  at  a  low  level.  At  first,  the  work  of 
all  card  punchers  received  100%  verification  or  correction.  Later  39% 
qualified  for  only  sample  verification. 

"Work  subject  only  to  sample  verification  flowed  through  the  process 
six  times  faster  than  otherwise.  Deming  and  Leon  Geoffrey,  in  an 
article  in  the  September  1941  issue  of  the  JOURNAL  OF  THE 
AMERICAN  STATISTICAL  ASSOCIATION,  estimated  that  the 
introduction  of  quality  control  saved  the  Bureau  $263,000,  which  was 
transferred  to  other  work,  and  also  contributed  to  earlier  publication. 
Use  of  statistical  quality  control  procedures  has  been  a  standard 
practice  at  the  Bureau  of  the  Census  ever  since." 

The  next  relevant  scenario  began  early  in  1942,  soon  after  war  was 
declared  against  the  United  States  by  Japan,  Germany,  Italy,  and  their 
allies.  By  then  Ed  Deming  was  well  established  at  the  Census  Bureau,  but 
was  also  working  half  time  as  Consultant  to  the  Secretary  of  War.  Recent 
Wilks  Award  winner,  W.  Allen  Wallis,  now  Undersecretary  of  State  for 
Economic  Affairs  and  then  on  the  faculty  of  Stanford  University,  tells  of 
those  times  in  the  June  1980  issue  of  the  JOURNAL  OF  THE  AMERICAN 
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STATISTICAL  ASSOCIATION: 


"The  atmosphere  there  that  spring  was  satirized  by  a  squib  in  the 
student  paper  saying,  'It  is  rumored  that  in  the  outside  world  there  is 
a  war  and  a  shortage  of  Coca  Cola.'  As  one  of  several  statisticians  - 
Holbrook  Working,  Eugene  Grant,  Quinn  McNemar,  Harold  Bacon  - 
seeking  some  way  that  we  at  Stanford  could  contribute  to  the  war 
sffort,  I  wrote  on  April  17,  1942,  to  a  friend  in  Washington,  W. 
Edwards  Deming  of  the  Census  Bureau,  that  'those  of  us  teaching 
statistics  in  various  departments  here  are  trying  to  work  out  a 
curriculum  adapted  to  the  immediate  statistical  requirements  of  the 
war.  It  seems  probable  that  a  good  many  students  with  research 
training  might  by  training  in  statistics  become  more  useful  for  war 
than  in  their  present  work,  or  might  increase  their  usefulness  within 
their  present  fields . ' 

"Deming  responded  on  April  24  with  four  single-spaced  pages  on 
the  letterhead  of  the  Chief  of  Ordnance,  War  Department.  After  some 
explanatory  background  on  the  theme  that,  'the  only  useful  function  of 
a  statistician  is  to  make  predictions,  and  thus  to  provide  a  basis  for 
action’,  he  wrote: 

'"Here  is  my  idea.  Time  and  materials  are  at  a  premium,  and  there 
is  no  time  to  be  lost.  There  is  no  royal  short  cut  to  producing  a 
highly  trained  statistician,  but  I  do  firmly  believe  that  the  most 
important  principles  of  application  can  be  expounded  in  a  very  short 
time  to  engineers  and  others.  I  have  done  it  and  have  seen  it  done. 
You  could  accomplish  a  great  deal  by  holding  a  school  in  the  Shewhart 
methods  some  time  in  the  near  future.  I  would  suggest  a  concentrated 
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effort  —  a  short  course  followed  by  a  long  course.  The  short  course 
would  be  a  two-day  season  for  executives  and  industrial  people  who 
want  to  find  out  some  of  the  main  principles  and  advantages  of  a 
statistical  program  in  industry.  It  would  be  a  sort  of  popularization, 
four  lectures  by  noted  industrial  people  who  have  seen  statistical 
methods  used  and  can  point  out  some  of  their  advantages.  The  long 
course  would  extend  over  a  period  of  weeks,  or,  if  given  evenings, 
over  a  longer  period.  It  would  be  attended  by  the  people  who  actually 
intend  to  use  statistical  methods  on  the  job.  In  many  cases  they 
would  be  delegated  by  the  men  who  had  attended  the  short  course.' 

'"I  would  suggest  that  both  courses  be  thrown  open  to  engineers, 
inspectors,  and  industrial  people  with  or  without  mathematical  and 
statistical  training.  Naturally,  any  person  who  has  had  considerable 
statistical  training  would  be  in  a  position  to  get  much  more  out  of  the 
course,  but  few  would  be  in  this  fortunate  position...' 

"On  May  1,  I  was  able  to  write  Deming  that,  'Your  letter  arrived  a 
few  hours  ago... The  specific  suggestions  struck  home  so  well  that 
Holbrook  Working  (Chairman  of  the  University  Committee  on  Statistics) 
has  already  talked  with  two  or  three  of  the  key  people  and  arranged  a 
general  meeting  of  everyone  in  statistics';  on  May  21  the  first  letter 
about  the  course  went  to  firms  supplying  Army  ordnance  in  the 
western  region;  and  the  first  course  was  given  in  July  1942  at 
Stanford." 

A  short  article  by  Holbrook  Working,  published  in  SCIENCE  in 
November,  1942  describes  this  effort.  Working,  after  some  preliminaries 
and  his  description  of  the  Deming  letter,  went  on  as  follows. 
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"The  suggestion  posed  two  problems:  that  of  providing  for  the 
requisite  instruction,  and  that  of  bringing  to  the  course  men  actually 
in  a  position  to  apply  the  methods. 

"Suitable  machinery  for  organizing  and  financing  the  suggested 
course  was  already  in  existence  in  the  engineering  science  and 
management  War  Training  Program,  financed  by  the  Office  of 
Education.  The  institutional  director  of  Vhe  program  at  Stanford  took 
up  the  plan  with  enthusiasm.  Aided  by  active  support  from  the 
Ordnance  Department,  through  its  San  Francisco  District  Office,  he 
was  able  to  bring  together,  in  early  July,  less  than  six  weeks  after 
the  original  suggestion  had  been  received,  a  group  of  twenty-nine  key 
men  from  industries  holding  war  contracts  and  from  procurement 
agencies  of  various  branches  of  the  armed  services.  These  men,  with 
three  others,  entered  upon  an  intensive  ten-day  course  with  classes 
running  eight  hours  a  day.  All  thirty-two  men  completed  the  course." 

Dr.  Working  went  on  to  describe  a  second  course,  offered  in  Los 
Angeles  in  September,  1942,  and  then  discussed  the  personnel  involved  in 
the  instruction. 

"Two  Professors,  Eugene  L.  Grant  and  Holbrook  Working  from 
different  departments  of  Stanford  University  and  Dr.  W.  Edwards 
Deming  from  the  Census  Bureau  worked  together  in  each  course.  A 
fourth  man  on  the  staff  for  each  course  was  drawn  from  industry  to 
present  the  point  of  view  of  a  man  meeting,  from  day  to  day,  the 
practical  problems  of  applying  the  methods  under  discussion. 


Dr.  Churchill  Eisenhart,  in  his  notes  on  Deming  accomplishments, 
describes  subsequent  events. 

"The  course  was  such  a  success  that  early  in  1943,  Working  was 
chosen  to  head  the  now  famous  major  national  program  that  put  on 
intensive  8-day  courses  in  statistical  quality  control  throughout  the 
country,  under  the  auspices  of  the  Office  of  Production  Research  and 
Development  of  the  U.S.  Office  of  Education.  Deming  was  the  teacher 
of  23  of  these  courses.  By  March  1945,  they  had  been  attended  by 
more  than  1900  persons  from  678  industrial  concerns  In  the  United 
States  and  13  in  Canada.  Many  of  the  'students'  in  the  earlier  of 
these  went  out  to  serve  as  'instructors'  in  part-time  courses  that 
brought  the  message  to  an  additional  31,000  persons  in  American  and 
Canadian  industry,  and  2500  persons  attended  other  part-time  courses 
in  statistical  quality  control.  The  program  had  an  enormously 
beneficial  effect  on  the  quality  and  volume  of  American  and  Canadian 
war  production;  and  'prepared  the  soil'  for  the  establishment  of  the 
American  Society  for  Quality  Control  (ASQC)  in  February  1946,  in  the 
founding  of  which  Deming  also  played  an  important  role.” 

Ed  Deming  agrees  that  he  did,  indeed,  play  an  important  role  in  the 
founding  of  the  ASQC. 

"Wherever  I  taught  I  told  the  people,  'Nothing  will  happen  if  you 
don't  keep  working  together.  And  you've  learned  only  a  little.  I 
know  only  a  little.  You  must  keep  on  working  and  meeting  together. 
Get  someone  to  send  out  postal  cards,  and  persuade  someone  to  let  you 
use  a  room  for  an  evening.'  And  they  did  it.  It  was  that  nucleus 
upon  which  congealed  the  ASQC." 


In  the  1981  MILITARY  SCIENCE  AND  TECHNOLOGY  interview,  Ed 
Deming  stated  in  a  few  words  the  principal  reason  that  the  brilliant 
successes  in  using  statistical  quality  control  methodology  to  increase  quality 
and  productivity,  later  to  be  exhibited  on  a  grand  scale  in  Japan,  were  not 
realized  in  this  country. 

"The  courses  were  well  received  by  engineers,  but  management  paid 
no  attention  to  them.  Management  did  not  understand  that  they  had  to 
get  behind  quality  control  and  carry  out  their  obligations  from  the  top 
down . " 

In  our  recent  conversations,  he  expanded  on  this  theme,  discussing  first 
the  random  variation  that  defines  a  process,  a  manufacturing  process,  say, 
in  an  industrial  setting. 

"In  the  wartime  courses  we  taught  people  that  there  is  variation  in 
all  things  and  that  the  readings  that  one  takes  from  a  manufacturing 
process  must  exhibit  stable  randomness,  or  they  don't  have  any 
meaning  as  far  as  defining  the  process.  Any  instabilities  can  help  to 
point  out  specific  times  or  locations  of  local  problems.  Once  these 
local  problems  are  removed,  then  there  is  a  process  that  will  persist 
until  somebody  changes  it. 

"Changing  the  process  is  management's  responsibility.  And  that  we 
failed  to  teach.  Professor  Working  thought  that  it  would  be  a  good 
idea  to  include  management  in  the  courses,  so  we  devoted  one 
afternoon  to  letting  the  people  bring  their  management.  Well,  some  did 
come,  but  most  did  not.  And  I  don't  think  we  had  anything  wonderful 
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to  tell  them.  We  had  no  stories  to  tell  them. 


"By  1950,  these  simple  methods  that  we  had  taught  were  working  all 
right,  but  nothing  astounding  happened.  Not  that  they  weren't 
accomplishing  something,  but  it  was  only  a  small  part  of  what  could  be 
accomplished.  The  big  gains  come  from  changes  in  the  system,  the 
responsibility  of  management. 

In  Japan,  management  did  take  responsibility  for  putting  statistical 
quality  control  methods  to  work.  The  story  of  how  that  happened  begins  in 
1946.  In  that  year.  Dr.  Deming  made  a  trip  around  the  world  under  the 
auspices  of  the  Economic  and  Scientific  Section  of  the  U.S.  Department  of 
War.  While  he  was  in  India,  working  with  Mahalanobls,  the  famous  Indian 
statistician  who  had  founded  the  prestigious  Indian  Statistical  Institute,  he 
got  orders  to  continue  on  to  Japan.  Kc  described  those  times  to  me. 

"I  stayed  In  Japan  for  two  months  to  assist  with  studies  of 
nutrition,  agricultural  production,  housing,  fisheries,  etc  In  that 
way  I  became  friends  with  and  learned  from  some  of  the  great  Japanese 
statisticians.  Statistics  was  well  established  in  Japan." 

He  is  not  aware  of  how  there  came  to  be  so  many  learned  statisticians  in 
Japan  those  many  years  ago,  but  he  remembered  that  a  Dr.  Seito  had  been 
studying  statistics  at  University  College  in  London  whan  he  was  there  a  few 
years  earlier. 

"In  1948,  I  went  again  to  Japan,  this  time  for  the  Department  of 
Defense,  to  do  more  of  what  I  had  done  before.  I  made  an  effort  to 
talk  whenever  possible  with  Japanese  statisticians.  I  would  go  to  the 
Post  Exchange,  where  I  had  privileges,  and  buy  food.  Then  I  would 


lug  it  to  the  Army  operated  Dai-lchi  Hotel  where  I  had  a  very  small 
room.  If  I  said  it  was  8  feet  by  9  feet,  I  wouldn't  be  far  off.  Then  I 
would  arrange  for  a  private  dining  room  in  the  hotel  and  serve  the 
food  to  my  Japanese  friends. 

"Any  food  tasted  good  to  them.  I'm  sure.  We'd  sit  around  the  table 
and  talk.  I  had  no  vision  of  what  was  to  happen.  I  just  told  them 
that  they  were  important  to  the  country  In  the  reconstruction  of 
Japan.  This  idea  was  new  to  them. 

"Now,  there  is  a  sub-plot  involving  a  Mr.  Ken-lchi  Koyanagi,  who 
had  earlier  been  in  jail  for  8  years  --  ostensibly  for  being  a 
Communist.  Whether  he  had  been  under  house  arrest  or  actually  in 
jail,  I  don't  know.  Probably  all  there  was  to  it  was  that  he  had  a 
mind  of  hie  own  and  wouldn't  go  along  with  the  war  lords,  i  say  this 
because  when  it  came  time  for  him  to  get  a  visa  later  to  come  to  this 
country,  there  was  no  great  problem. 

"His  major  in  the  university  was  German  literature.  Most  people 
who  rise  in  management  in  Japan  never  have  studied  Management 
Science,  thank  goodness.  It's  better  that  they  don't. 

"In  1947  he  formed  the  Union  of  Japanese  Scientists  and  Engineers 
(JUSF.)  consisting  then  of  about  7  men,  their  purpose  being  the 
reconstruction  of  Japan.  Mr.  Koyanagi  held  the  group  together.  And 
Dr.  Nishibori,  who  was  in  the  original  group  and  later  Chairman  of 
Japan's  equivalent  of  our  Atomic  Energy  Commission,  told  me  that  they 
had  nothing  much  to  talk  about.  They  would  just  eat  and  drink. 
Suddenly  one  night,  Dr.  Nishibori  had  the  bright  idea  that  statistical 
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methods  could  help  in  the  reconstruction  of  Japan.  This  would  be  a 
way  of  helping  that  wouldn't  require  new  equipment,  which  they  had 
no  means  of  obtaining. 

"One  of  the  principal  problems  of  Japanese  industry  at  that  time 
was  that  the  captive  markets  of  China  and  Korea  that  they  had  had 
prior  to  the  war,  were  now  lost  to  them.  And  they  needed  to  trade  so 
that  they  could  import  food. 

"Came  in  1949  a  letter  asking  me  to  teach  statistical  methods  in 
industry.  I  couldn't  go  at  that  time,  though  I  wished  to.  I  had  too 

many  projects  going,  so  I  kept  stalling.  I  finally  did  go  In  June  of 

1950  under  the  auspices  of  the  Supreme  Commander  of  Ailed  Powers." 

THE  MAIN  I  CHI  DAILY  NEWS  OF  TOKYO,  on  the  occasion  of  the 
presentation  of  the  Darning  prizes  on  November  10,  1965,  described  the  visit 
and  the  conditions  in  Japan  immediately  following  the  war. 

"The  scholastic  contact  between  Japan  and  Dr.  Doming  dated  back 
to  April  1950  whan  Ken-ichl  Koyanagi  managing  director  of  the  JUSE, 
wrote  to  Dr.  Darning,  then  in  the  U.S.  asking  for  lectures  on 
statistical  quality  control  when  he  visited  Japan  later  in  the  year.  He 
readily  accepted  the  plea. 

"At  that  time,  few  Japanese  realized  the  significance  of  quality 
control.  In  the  prewar  years,  there  were,  indeed,  some  Japanese 
scholars  and  engines  s  who  wore  engaged  in  the  study  of  quality 

control,  and  some  of  them  attempted  to  put  it  Into  practice.  But  no 

company  dared  to  carry  out  the  wholesale  introduction  of  the 
revolutionary  Idea. 


"After  the  war,  the  nation's  industry  was  quick  to  rise  again,  but 
the  quality  of  its  products  were  all  but  inferior.  Faced  with  enormous 
demand,  manufacturers  were  all  busy  in  turning  out  as  many  products 
as  possible,  and  no  one  cared  about  quality. 

"The  concept  of  quality  control  made  inroads  Into  the  Japanese 
industries  in  the  form  of  an  Occupation  Forces  order  to  communication 
equipment  manufacturers.  When  thoy  started  to  employ  the  modern 
production  formula,  some  private  organizations  paid  a  deep  concern. 
Soon  they  stepped  into  the  field  and  started  dissemination  activities. 

"Independent  from  these  organizations,  the  JUSE  also  launched  an 
educational  service  of  quality  control  In  1948.  A  series  of  lectures  was 
sponsored  on  the  subject  of  statistical  analysis  of  small  samples. 
Several  Japanese  experts  gathered  to  form  a  research  group,  primarily 
aimed  at  collecting  necessary  literature.  But  these  activities  had  a 
discouraging  result;  there  was  little  experience  and  materinl  available. 
Still  under  occupation,  Japan  was  in  no  position  to  obtain  enough 
literature  and  material  related  to  quality  control. 

"Then  came  the  offer  from  Dr.  Darning  to  the  joy  and  surprise  of 
all  the  people  concerned.  In  his  first  lecture  meeting  in  Tokyo  in 
mid-1950,  230  scholars  and  statisticians  gathered,  impressed  by  the 
exciting  concept  of  statistical  quality  control  uttered  by  the  U.S. 
scholar.  In  another  lecture  meeting  in  Fukuoka,  110  were  present. 

"Dr.  Deming  called  on  the  students  to  come  out  of  their  studies 
and,  with  courage  and  confidence,  go  into  factories,  to  keep  contact 


with,  and  teach,  business  managers  and  engineers,  and  to  promote 
their  theoretical  research  on  the  application  of  statistical  methods" 

He  recalls: 

"i  lectured  in  English,  but  I  had  a  wonderful  translator,  Mr. 

Hisamachi  Kano.  His  father  was  a  banker,  and  as  a  child,  he  lived  in 
New  York,  London  and  Paris,  so  he  learned  English  and  French  as  he 
was  growing  up.  He  probably  learned  Japanese  at  home." 

"His  English  was  absolutely  perfect,  with  every  kind  of  idiom.  I 
was  very  fortunate  because  I  had  him  with  me  for  the  duration  of 
every  visit  for  o  period  of  over  ten  years. 

Dr.  Deming  described  to  me  the  fateful  events  that  Involved  Japanese 
higher  management  in  the  educational  process  and  provided  the  critical 

Impetus  for  changing  the  Image  of  Japanese  products. 

"They  were  wonderful  students,  but  on  the  first  day  of  the 
lectures  a  horrible  thought  came  to  me,  'Nothing  will  happen  in  Japan; 
it'll  be  a  farce  unless  I  talk  to  top  management.'  By  that  time  I  had 
some  idea  of  what  top  management  must  do.  There  are  many  duties  to 
be  performed  that  only  the  top  people  can  do:  consumer  research  for 
example,  work  with  vendors  just  for  example.  I  knew  that  I  must 

reach  top  management.  Otherwise  it  would  Just  be  another  flop  as  it 

was  in  the  states. 

I  Immediately  talked  to  American  friends  who  knew  the  right 
Japanese  and  before  long,  I  was  talking  to  Mr.  Icharo  Ishikawa,  who 
had  formed  the  great  Kei-dan-ren,  the  Japanose  association  of  top 
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management. 


"I  had  3  sessions  with  Mr.  Ishikawa;  and  at  the  end  of  the  third 
session,  he  understood  what  I  needed  to  do.  He  sent  telegrams  to  the 
45  or  so  top  level  men  to  come  to  the  Industry  Club  the  next  Tuesday 
at  5  o'clock  to  hear  Or.  Deming.  And  they  all  came. 

"I  did  the  best  I  could.  I  gave  them  encouragement.  That  was  the 
main  thing.  I  told  them  that  they  could  produce  quality  for  the 
consumer,  partly  industrial,  partly  household,  for  the  Western  world, 
in  return  for  food.  Conditions  were  such  that  they  would  have  to  do 
that. 

"They  thought  that  they  could  not  because  they  had  such  a  terrible 
reputation  when  it  came  to  quality.  But  they  knew  what  good  quality 
was.  Ask  anybody  in  our  Navy,  and  they'll  tell  you  that.  What  they 
made  for  military  purposes  was  superb.  But  for  consumer  goods, 
they'd  never  tried.  They  didn't  know  what  It  was  to  stand  back  of 
any  goods.  At  that  time  a  Japanese  Item  wouldn't  last  very  long; 
there  was  no  endurance. 

"I  told  them,  'Those  days  are  over.  You  can  produce  quality. 
You  hava  a  method  for  doing  it.  You've  learned  what  quality  is.  You 
must  carry  out  consumer  research,  look  toward  the  future  and  produce 
goods  that  will  have  a  market  years  from  now  and  stay  in  business. 
You  have  to  do  it  to  eat.  You  can  send  quality  out  and  get  food 
back.  The  city  of  Chicago  does  it.  The  people  of  Chicago  do  not 
produce  their  own  food.  They  make  things  and  ship  them  out. 
Switzerland  does  not  produce  all  their  own  food,  nor  does  England.'" 
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"Incoming  materials  were  terrible,  oft  gauge  and  off  color,  nothing 
right.  And  I  urged  them  to  work  with  the  vendors  and  to  work  on 
instrumentation.  A  lot  of  what  I  urged  them  to  do  came  very  naturally 
to  the  Japanese,  though  they  were  not  doing  it.  I  said,  'You  don't 
need  to  receive  the  junk  that  comes  in.  You  can  never  produce 
quality  with  that  stuff.  But  with  process  controls  that  your  engineers 
are  learning  about,  specifications  as  loose  as  possible,  consumer 
research,  redesign  of  products,  you  can.  Don't  just  make  it  and  try 
to  sell  It.  But  redesign  it  and  then  again  bring  the  process  under 
control.  The  cycle  goes  on  and  on  continually,  with  ever  Increasing 
quality.' 

"I  knew  the  problems  because  I'd  been  at  Aberdeen  Proving 
Ground,  working  there  for  the  War  Department,  with  people  in 
industry.  And  look  at  the  Census  Bureau.  It  was  one  of  the  largest 
organizations  to  be  Immersed  in  quality. 

"One  of  the  big  problems  of  management  is  to  define  quality  and 
realize  that  there  are  several  facets.  One  is  what  you're  trying  to  do 
for  the  future,  whatever  quality  you’re  aiming  at.  Should  your 
purchasing  agent  continue  to  buy  this  kind  of  paint,  or  should  he 
switch?  But  also,  how  about  turning  out  product  today?  What  is  the 
plant  manager's  job  today? 

"Now  only  the  management  can  work  on  that  problem  of  defining 
quality.  It's  a  complicated  problem  with  no  easy  solutions,  but  it's 
management's  responsibility. 

"I  tried  to  explain  these  things  to  them,  and  apparently  they 
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understood.  They  wanted  more  conferences,  so  we  had  more.  It  was 
a  terrifying  experience  for  me  because  I  was  new  at  it.  I  was  a 
technical  man. 

"I  told  them  they  would  capture  markets  the  world  over  within  5 
years.  They  beet  that  prediction.  Within  4  years  buyers  all  over  the 
world  were  screaming  for  Japanese  products. 

"I  was  back  in  Japan  in  6  months,  in  Janury  of  1951.  They 

already  had  had  many  brilliant  successes,  brilliant  fires,  just  as  they 
had  had  here  during  the  war.  But  that's  not  quality;  those  are  just 
dividends.  The  top  management  showed  me  what  they  were  doing. 
Mr.  Nishimura,  President  of  the  Furukawa  Electric  Company,  was 

himself  working  to  evaluate  the  process  that  produced  insulated  wire. 
He  brought  control  charts  to  show  me,  and  ha  was  able  to  reduce  the 
amount  of  rework  to  10%  of  what  it  had  been. 

Mr.  Tanabe,  President  of  the  Tanabe  Pharmaceutical  Company,  was 

working  himself  in  quality  control.  In  a  few  months  he  was  producing 

3  times  as  much  para  aminosalyscilic  acid  as  before,  with  the  same 
machinery,  by  just  improving  the  system. 

"But  you  cannot  improve  the  system  until  you've  achieved  statistical 
control.  Then  engineers  and  chemists  can  see  that  it  will  stay  this 
way  until  they  make  some  changes. 

"Now  six  months  later  here  wore  these  members  of  top  management 
showing  me  what  they  had  done.  Six  months  after  that  trip,  I  was 
there  again,  and  a  year  later  there  again.  They  were  working  hard, 
and  they  were  getting  results.  I  made  it  clear  to  them  in  those  first 
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conferences  that  this  must  be  company  wide.  'Everybody  in  the 
company  has  a  job  to  do  to  improve  quality.  And  as  you  improve 
quality,  your  productivity  will  go  up.  Vour  customers  will  be  happy, 
and  you'll  have  something  to  sell.' 

"I  also  told  them  'This  movement  must  be  nationwide.  You  must 
teach  other  companies,  teach  your  competitors,  move  along  together. 
As  you  learn, » tell  others.'  I  didn't  have  to  tell  them  that.  That  was 
the  natural  Japanese  way  of  working  But  I  did  tell  them  anyway. 

"By  the  time  I'd  made  several  trips  to  Japan,  Juse  was  able  to 
teach  hundreds  of  people.  They  had  courses  for  people  outside  of 
Tokyo  in  the  daytime  and  courses  In  Tokyo  In  the  evening  for  people 
who  were  working  there  during  the  day.  There  were  also  courses  for 
management.  They  trained  almost  20,000  engineers  in  rudimentary 
statistical  methods  within  10  years.  These  courses  today  are  booked 
up  seven  months  ahead. 

Clearly,  the  Japanese  appreciate  what  Ed  Darning  and  statistical  quality 
control  have  done  to  change  their  destiny.  The  MAINICHI  DAILY  NEWS 
describes  the  history  of  the  Doming  Prize,  which  symbolizes  this 
appreciation. 

"The  Deming  Prize  was  created  in  1951  by  the  Union  of  Japanese 
Scientists  and  Engineers  (JUSE)  to  commemorate  the  friendship  and 
contribution  of  Dr.  Deming  to  the  whole  spectrum  of  Japanese 
industry.  The  prize  has  played  a  significant  role  to  give  an  impetus 
to  industry  in  its  dazzling  growth. 


"The  Deming  Prize  is  a  silver  medal.  Designed  by  Professor 


Kiyoshi  Unno  of  Tokyi  University  of  Fine  Art*  and  some  other  artists, 
the  medal  bears  an* engraved  profile  of  Dr.  Doming. 

"The  Doming  Prize  is  divided  into  three  categories.  The  prize  for 
research  and  education  is  awarded  to  those  who  made  excellent 
researches  in  theory  and  application  of  quality  control.  Another  prize 
for  application  is  given  to  corporations  or  plants  which  attained 
outstanding  results  in  practicing  quality  control.  The  third  prize  is 
provided  for  smaller  enterprises. 

"The  prize  has  been  awarded  annually  ever  since  1951.  The 
Doming  Prize  Committee  i3  responsible  for  the  selection  of  the  winners 
from  among  a  number  of  candidates.  Parallel  with  the  progress  in 
Japan  in  the  concept  of  quality  control,  the  selection  standard  has 
been  rising  year  after  year,  and  the  race  for  the  laurels  has  become 
keen.  It  is  said  that  most  corporate  candidates  are  spending  years  in 
streamlining  and  reinforcing  their  quality  control  setup  under  the 
guidance  of  specially  invited  experts  before  they  apply  for  the  prize." 

Business  Week,  on  page  21  of  a  special  advertising  section  on  "Japan: 
Quality  Control  and  Innovation"  of  July  20,  1981,  lists  the  winners  of  the 
Deming  Prize  for  Application  for  the  years  1954  through  1980  and  discusses 
its  impact. 


"Each  year  the  competition  grows  in  intensity,  as  more  and  more 
companies  volunteer  to  undergo  the  close  scrutiny  required.  For  the 
firm  that  wins  the  Prize,  and  those  that  gain  one  of  the  associated 
awards,  however,  the  rewards  are  significant,  in  profits  as  well  as 
prestige.  For  other  companies,  the  ceremony  is  a  time  for  self¬ 


reckoning.  The  innovations  in  quality-control  honored  in  any  year 


usually  soon  become  national  norms. 
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ANALYSIS  OF  FERROGRAPHIC  ENGINE  WEAR 
DATA  USING  QUALITY  CONTROL  TECHNIQUES 


Robert  L.  Launer  and  Edward  A.  Salbel 
U.S.  Arqy  Research  Office 
Research  Triangle  Park,  N.C.  27709 


1.  Background. 

'7^ It  Is  generally  accepted  that  wear  :s  the  leading  factor  In  engine  and 
gear  failure.  There  are  many  types  of  wear,  some  of  which  are:  adhesion, 
abrasion,  corrosion,  erosion,  frettlug,  cavitation,  fatigue,  melting,  ablution 
and  del  ami  nation.  Each  of  these  results  In  Its  characteristic  form  of  wear 
particle,  the  Identification  of  which  Is  sometimes  difficult.  There  are  many 
methods  for  Indentlfylng  these  particles  and  for  monitoring  their  development 
over  time.  One  such  method  Is  ferrography. 

Ferrography  Is  a  technique  developed  by  Seifert  and  Wescott  for  separating 
wear  particles  from  the  lubricant  matrix  and  depositing  these  on  a  glass 


slide,  arranged  or  sorted  by  particle  size  [4,  7](.  This  slide  Is  then 
examined  microscopically.  An  Indirect  measure  of  wear  Is  obtained  by 
measuring  the  amount  of  light  which  Is  transmitted  through  the  glass  slide, 
subject  to  the  aaxiunt  of  particles  which  have  been  deposited.  The 
transmittance  Is  reported  as  the  percentage  of  the  area  within  the  field  of 
view  which  Is  covered  by  the  deposited  particles.  Measurements  are  made 
^reas  on  the  slide  corresponding  to  the  large  particles  and  to  the/^mall 
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particles.  The  two  measurements  are  called  by  workers  In  the  field, 
respectively.  The  particles  are  deposited  by  dripping  the  engine  or 
transmission  oil  onto  an  Inclined  glass  slide  which  Is  Immersed  In  a  magnetic 
field.  The  larger  particles  are  thus  deposited  first  and  the  smallest 


particles,  last.^A  good  survey  of  this  method  Is  presented  In  [6], 
2. 


Statement  of  Objective.  — — .  — 

Our  objective  Is  to  produce  an  easy  to  use  method  for  Improving  the  amount 
of  Information  which  can  be  obtained  In  Ferrography  without  an  Increase  In 
time,  effort  and  Instrumentation.  As  things  stand  now,  optical  measurements 


are  made  from  the  ferrogram  deposit  and  an  Index  of  wear  severity,  Ie  Is 


calculated  using  an  arbitrary  relationship 


where  A^  Is  percent  area  covered  by  particles  at  the  entry  deposit,  (particles 
greater  than  5  ym)  and  A  Is  the  percent  area  covered  by  particles  at  50  mm 
from  the  exit  of  the  Ferrogram,  (particles  ranging  from  1  to  2  ym.)  [6].  This 
Index  of  severity,  proposed  by  V.  Wescott,  Is  attractive  because  of  Its 
conciseness  and  the  ease  with  which  It  Is  calculated.  Since  It  contains  only 
Information  obtained  directly  from  the  Ferrogram,  this  measure  Is  apparently 
germane  and  relevant. 

3.  Brief  Discussion  of  Current  Methodology. 

As  a  direct  measure  of  wear,  I$  Is  difficult  to  Interpret.  Let  I$,  A$  and 
Al  represent  the  time  derivatives  of  Is,  A$,  and  A^.  Notice  that, 


31. 


IE 


■  -2As<o,  and 


al. 


IE 


»  2A^  >o. 


•s  -  L 

•  •  • 

so  that  I$  ■  2A|_Al  -  2AgAg.  Therefore,  a  net  positive  change  In  I$  can  result 
from  either  an  Increase  In  AL  or  a  decrease  In  A$.  In  general,  simultaneous 
Increases  and/or  decreases  In  AL  and  A$  In  differing  amounts  may  result  In 
either  an  Increase  or  a  decrease  In  Is>  In  the  following,  we  propose  a  change 
In  this  Index  which  will  produce  a  direct  measure  of  the  ferrogram  Information 
which  Is  easy  to  compute  and  Interpret. 

A  Ferrogram  Is  an  Indirect  measure  of  engine  wear  at  a  specific  time  so 
that,  for  practical  purposes,  It  can  be  considered  a  monitoring  process.  The 
onset  of  failure  Is  signalled  by  a  fairly  abrupt  Increase  In  or  As  or  both. 

Early  failures  are  Indicated  by  premature  deviations  from  the  normal  values  or 
trends  In  one  of  these  parameters.  It  would  be  very  useful  to  devise  a  scale 
for  plotting  Ferrogram  values  with  automatic  warning  limits  so  that 
Interpretation  of  Individual  cases  could  be  reduced  to  a  minimum.  If  this 
were  accomplished  with  a  preliminary  sample  or  other  past  history  (such  as 
factory  test  data)  to  establish  benchmarks,  we  will  have  described  a  quality 
control  monitoring  process. 

Ferrogram  measurements  exhibit  unpredictable  variation  which  demands  a 
statistical  analysis  for  proper  Interpretation.  Although  the  statistical 
distributions  of  A^  and  A$  are  somewhat  normal  In  appearance  [5],  we  suggest 
that  several  repeated  measurements  be  taken  of  each  value  from  each  Ferrogram, 
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yielding  average  XL  and  As,  so  that  the  assumption  of  normality  may  be 
justified  by  Invoking  the  central  limit  theorom  [3].  Since  these  measurements 
are  taken  from  the  same  Ferrogram,  there  Is  the  possibility  of  correlation 
between  them.  AL  and  A$  are,  however,  related  to  the  extreme  values  of  the 
available  measurements  from  the  Ferrogram  and  are,  therefore,  related  to  the 
extreme  order  statistics.  Since  the  extreme  order  statistics  are 
asymptotically  independent,  [2],  and  X$  are  assumed  to  be  Independent.  We 
will,  nevertheless,  present  a  method  which  will  allow  for  correlation  between 
them. 

4.  Proposal  for  an  Improved  Method. 

Let 

x  «  sample  average,  AL,  at  time  t 
y  -  sample  average,  A$,  at  time  t 
ux(t)  ■  expected  value  of  AL  at  time  t 
Uy(t)  «  expected  value  of  Ag  at  time  t 

o2  ■  variance  of  AL 

o'-  ■  variance  of  A 

/  5 

p  -  correlation  between  A.  and  A„ 

L  S 


Under 
x  and 


the  assumptions  stated  previously,  the  joint  statistical  distribution  of 
y  Is 

f(x,y)  ■  Uirc^yVwTr1  exp  [-g(x,y)/2(l-P2)]  (2) 


where 


X-y  9  j-y.,  « 

g(x,y)  -  (— ■  )2-  2p  (-rr*)  +  b-^)2  • 


X-Vy. 


y-y, 


(3) 


The  appropriate  regions  for  monitoring  sample  values  (x,y)  are  the  ellipses  of 
equal  probability  density,  for  the  probability 

(4) 


a  -  exp  M, 


5 


(6) 


where  A  1$  the  region  enclosed  by  the  ellipse  LI 3. 
g(x,y)  ■  a2  (1-p2)  . 

In  order  to  standardize  the  graphical  representation  of  the  sample 
values,  It  Is  recommended  that  the  ellipses  (  6 )  be  transformed  to  unit  circles 
as  follows.  When  x  and  y  are  independent  the  ellipse  (  6  )  becomes 


Let 


ar  ■  U-ux)/ox  and  as  -  (y-uy)/ay 


(8) 


Then  (7)  becomes 


r2  +  s2  «  1 


(9) 


The  new  Index  of  severity,  Is  made  by  transforming  the  data  thus: 


Then, 


and 


j  ■  u2  +  v2 
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(10 

(11 


Is  the  proposed  new  Index  of  severity.  This  value  should  be  compared  to  an 
extreme  upper  tall  percentage  point  of  the  central  chi-square  distribution 
with  2  degrees  of  freedom.  For  example,  the  probabilities  >.01  and  .005 
correspond  to  the  values  9.2  and  10.6,  respectively.  If  there  Is  correlation 
present  these  values  are  reduced,  with  the  lower  bounds  6.6  and  7.9 
corresponding  to  perfect  correlation.  On  the  other  hand,  the  onset  of  failure 
Is  marked  by  Instability  of  the  distribution  of  the  particle  sizes  with  AL,  A$ 
or  both  rapidly  becoming  very  large,  depending  on  the  underlying  cause  of  the 


failure.  This  Implies  that  these  measures  change  separately  or  Independently, 
so  that  the  deflation  of  these  values  due  to  correlation  would  tend  to 
emphasize  them  during  the  onset  of  failure.  The  appropriate  critical  values 
In  either  case,  then,  could  be  obtained  from  the  chi-square  distribution  with 
2  degrees  of  freedom.  Me  suggest  using  the  value  10  (or  9  If  the  user  Is 
conservative)  for  the  critical  value  of  J^. 

5.  A  Numerical  Example. 

The  foregoing  development  might  appear  somewhat  complicated,  although 
Is  only  slightly  more  complicated  than  (1).  We  maintain  that  J$  contains  more 
engine  history  and  therefore  more  Information  on  which  to  base  automated 
decisions.  We  further  suggest  that  Js  and  the  related  preceding  formulation 
can  be  easily  computed  with  a  handheld  computer  or  even  programmed  for  a 
microcomputer.  The  following  example  will  Illustrate  the  point. 

Suppose  that  It  has  been  determined  that  a  certain  helicopter  engine  Is 
characterized  by  ^(t)«  15  +  .00625t,  uy(t)»6+.003t,  o^-tlVl,  Oy-2.75,  and  t  is 
engine  operation  time  In  hour*.  Suppose  further  that  the  engine  Ferrogram 
measurements  at  600  ,  650,  and  700  hours  are(^,  A$)«  (24.8,  6.1)  (24.1,8.6), 
and  (25.7,  12.7).  First  we  note  that  uJ600)»18.75  and  p  (600)«7.8.  Then 


The  other  values  of  and  the  values  of  I$  are  calculated  similarly  and  are 
given  In  the  table  below.  Notice  that  while  the  successive  values  of  I_ 
decrease,  the  third  value  of  Js  exceeds  the  critical  value,  which  Is  a  signal 
of  Impending  failure. 


4.35 

2.43 

11.28 
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